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Preface 


The  Department  of  Homeland  Security  is  responsible  for  protecting 
the  United  States  from  terrorism  through  prevention,  preparedness, 
and  response.  In  part,  this  goal  is  achieved  through  allocation  of  re¬ 
sources  to  states  and  urban  areas.  The  Urban  Areas  Security  Initiative 
(UASI)  is  a  Department  of  Homeland  Security  (DHS)  grant  program 
designed  to  enhance  security  and  overall  preparedness  to  prevent,  re¬ 
spond  to,  and  recover  from  acts  of  terrorism.  This  program  provides 
financial  assistance  to  address  unique  planning,  equipment,  training, 
and  exercise  needs  of  large  urban  areas  (DHS,  2004).  Although  many 
stakeholders  apparently  agree  that  these  allocations  should  reflect  the 
magnitude  of  risks  to  which  different  areas  are  exposed,  no  consensus 
has  emerged  on  how  this  might  be  accomplished. 

This  monograph  examines  several  challenges  to  risk-based  allo¬ 
cation  of  homeland  security  resources.  There  is  not  a  consistent  and 
shared  definition  of  terrorism  risk.  Estimating  terrorism  risk  requires 
treatment  of  numerous,  large  uncertainties.  There  is  no  existing 
framework  for  selecting  and  combining  risk  indicators.  Finally,  little 
work  has  been  directed  toward  methods  for  testing  how  the  accuracy 
and  distribution  of  risk  from  different  estimates  change  with  respect 
to  a  wide  range  of  assumptions  about  terrorist  threats  and  capabilities 
and  the  dearth  of  information  about  how  security  investments  might 
reduce  terrorism  risk.  This  monograph  addresses  each  of  these  issues 
and  proposes  solutions  to  all  except  the  final  one,  understanding  the 
relationship  between  investment  and  risk  reduction,  which — though 
a  critical  problem — has  been  left  for  further  study. 


VII 


viii  Estimating  Terrorism  Risk 


This  research  on  the  general  issue  of  how  to  estimate  terrorism 
risk  is  meant  to  inform  resource  allocation.  It  is  not  a  direct  assess¬ 
ment  of  current  DHS  practice  or  endorsement  of  insurance  risk  mod¬ 
eling.  Similarly,  it  does  not  represent  government  policy  and  is  not 
presented  as  such.  Rather,  it  is  intended  to  add  information  and  per¬ 
spective  to  the  rapidly  maturing  issue  of  risk-based  resource  allocation 
and  promote  discussion.  Further,  this  monograph  presents  some  data 
and  descriptions  of  DHS  processes  from  fiscal  year  2004  budget 
planning.  The  Department’s  use  of  risk  to  inform  resource  allocations 
may  have  since  changed,  though  details  of  these  processes  have  not 
been  publicly  disclosed. 

This  monograph  should  be  of  interest  to  federal,  state,  local,  and 
private  sector  officials  responsible  for  estimating  terrorism  risks  and 
providing  guidance  on  resource  allocation  and  prioritization  based 
upon  these  risk  estimates. 

This  study  results  from  the  RAND  Corporation’s  continuing 
program  of  self-initiated  research.  Support  for  such  research  is  pro¬ 
vided,  in  part,  by  donors  and  by  the  independent  research  and  devel¬ 
opment  provisions  of  RAND’s  contracts  for  the  operation  of  its  U.S. 
Department  of  Defense  federally  funded  research  and  development 
centers. 
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Summary 


The  Department  of  Homeland  Security  (DHS)  is  responsible  for  pro¬ 
tecting  the  United  States  from  terrorism  through  prevention,  prepar¬ 
edness,  and  response.  In  part,  this  goal  is  achieved  through  allocation 
of  resources  to  states  and  urban  areas.  The  Urban  Areas  Security  Ini¬ 
tiative  (UASI)  is  a  DHS  grant  program  designed  to  enhance  security 
and  overall  preparedness  by  addressing  unique  planning,  equipment, 
training,  and  exercise  needs  of  large  urban  areas  (DHS,  2004).  Al¬ 
though  there  appears  to  be  agreement  among  many  stakeholders  that 
these  allocations  should  reflect  the  magnitude  of  risks  to  which  differ¬ 
ent  areas  are  exposed,  no  consensus  has  emerged  on  how  this  might 
be  accomplished.  Indeed,  the  UASI  grant  program  has  frequently 
been  criticized  for  inadequately  calculating  risk  and  therefore  for 
failing  to  distribute  resources  in  proportion  to  urban  areas’  shares  of 
total  terrorism  risk. 

Ultimately,  efficient  allocation  of  homeland  security  resources 
would  be  determined  based  upon  assessment  of  the  cost  effectiveness 
of  alternative  risk-reduction  opportunities.  After  potentially  first  ad¬ 
dressing  obvious  and  easily  mitigated  risks,  this  requires  understand¬ 
ing  the  cost  effectiveness  of  different  types  and  amounts  of  invest¬ 
ment.  Neither  the  methods  nor  the  data  are  available  to  answer 
questions  about  the  effectiveness  of  available  risk-reduction  alterna¬ 
tives  or  to  determine  reasonable  minimum  standards  for  community 
preparedness.  Until  these  questions  are  answered,  allocating  home¬ 
land  security  resources  based  on  risk  is  the  next  best  approach  since 
areas  at  higher  risk  are  likely  to  have  more  and  larger  opportunities 
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for  risk  reduction  than  areas  at  lower  risk.  That  is,  resources  would  be 
allocated  roughly  proportionally  to  the  distribution  of  risk  across  ar¬ 
eas  receiving  funding. 

This  monograph  offers  a  method  for  constructing  an  estimate  of 
city  risk  shares,  designed  to  perform  well  across  a  wide  range  of  threat 
scenarios  and  risk  types.  It  also  proposes  and  demonstrates  a  frame¬ 
work  for  comparing  the  performance  of  alternative  risk  estimates 
given  uncertainty  in  measuring  the  elements  of  risk. 


Components  of  Risk 

Terrorism  risk  can  be  viewed  as  having  three  components:  the  threat 
to  a  target,  the  target’s  vulnerability  to  the  threat,  and  the  consequences 
should  the  target  be  successfully  attacked.  People  and  organizations 
represent  threats  when  they  have  both  the  intent  and  capability  to 
damage  a  target.  The  threats  to  a  target  can  be  measured  as  the  prob¬ 
ability  that  a  specific  target  is  attacked  in  a  specific  way  during  a 
specified  period.  Thus,  a  threat  might  be  measured  as  the  annual 
probability  that  a  city’s  football  stadium  will  be  subject  to  attack  with 
a  radiological  weapon. 

Vulnerability  can  be  measured  as  the  probability  that  damage  oc¬ 
curs,  given  a  threat.  Damages  could  be  fatalities,  injuries,  property 
damage,  or  other  consequences;  each  would  have  its  own  vulnerabil¬ 
ity  assessment.  Consequences  are  the  magnitude  and  type  of  damage 
resulting,  given  a  successful  terrorist  attack.  Risk  is  a  function  of  all 
three  components:  threat,  vulnerability,  and  consequences.  These 
constructs  can  be  used  to  measure  risk  consistently  in  terms  of  ex¬ 
pected  annual  consequences.  More  detailed  definitions  of  vulnerabil¬ 
ity,  threat,  and  risk  and  discussions  of  measures  for  each  are  presented 
in  this  monograph. 
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Uncertainty  and  Value  Judgments  in  Terrorism  Risk 
Assessment 

There  are  two  important  sources  of  uncertainty  in  estimating  terror¬ 
ism  risk.  The  first  includes  variability  and  error  in  estimates  of 
threats,  vulnerabilities,  and  consequences.  The  second  involves  how 
we  should  value  different  types  of  consequences.  Part  of  an  informed 
discussion  of  homeland  security  policy  rests  on  an  understandable 
and  transparent  means  of  accounting  for  uncertainties  in  estimates 
and  the  consequences  of  using  alternative  values. 

When  facing  uncertainty  about  estimates  and  values,  policy 
analysis  often  relies  on  best  estimates,  even  when  they  have  a  low 
probability  of  being  correct — and  a  high  probability  of  being  wrong. 
While  this  allows  us  to  generate  a  very  precise  estimate  of  risk,  in  the 
end,  if  the  estimates  poorly  represent  what  actually  happens  in  real 
life,  the  precision  is  misplaced.  So,  rather  than  seek  an  optimal 
method  for  estimating  risk,  we  seek  a  method  that  leads  us  to  make 
the  least  egregious  errors  in  decisionmaking  across  the  range  of  possi¬ 
ble  scenarios  that  might  develop  in  the  future.  Following  methods  of 
adaptive  planning  under  deep  uncertainty  (Davis  1994,  2002;  Lem- 
pert,  Popper,  and  Bankes,  2003),  we  seek  a  method  for  estimating 
risk  that  is  robust  because  it  has  the  lowest  expected  error  when 
evaluated  against  a  wide  range  of  possible  futures. 

One  approach  for  developing  an  estimate  with  these  properties 
would  be  first  to  define  multiple  sets  of  threats,  vulnerabilities,  and 
consequence  measures  and  use  them  as  the  basis  for  constructing  a 
single  risk  estimate.  Then  using  these  multiple  estimates,  develop  a 
single  description  of  how  risk  is  distributed  that  balances  across  mul¬ 
tiple  perspectives  of  terrorism  risk. 

Generating  multiple  risk  estimates  can  provide  plausible  bounds 
on  the  magnitude  of  terrorism  risk  estimates  and  how  different 
stakeholders  may  be  affected  based  on  where  they  live  or  what  out¬ 
comes  they  value  most.  From  several  estimates,  one  can  ask  how  low 
or  high  terrorism  risk  may  be  in  a  specific  city,  what  a  best  estimate  of 
risk  is  given  the  range  of  estimates  available,  and  how  answers  to  these 
questions  differ  when  considering  different  types  of  outcomes.  The 
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challenge  to  analysts  defining  a  single  picture  of  how  risk  is  distrib¬ 
uted  is  to  do  so  without  losing  significant  information.  This  requires 
specifying  how  to  deal  with  the  challenges  of  aggregation  given  both 
inherent  uncertainties  and  value  choices. 

Uncertainty  in  terrorism  risk  estimates  suggests  the  need  to  de¬ 
vise  means  of  hedging  our  homeland  security  policies  against  a  range 
of  distributions  of  risk  that  are  plausible  given  what  we  know  about 
uncertainties  in  our  risk  estimation  procedures.  So,  rather  than  seek 
an  optimal  method  for  estimating  risk,  we  seek  a  method  that  leads  us 
to  make  the  least  egregious  errors  in  decisionmaking  across  the  range 
of  possible  scenarios  that  might  develop  in  the  future.  This  presents  a 
problem  comparable  to  that  of  forecasting  economic  trends  using 
multiple  estimates  or  models  discussed  by  Clemen  (1989).  This  lit¬ 
erature  highlights  two  objectives  to  consider  when  combining  esti¬ 
mates:  1)  use  information  contained  in  the  multiple  estimates  to  im¬ 
prove  forecasting  accuracy  and  2)  make  note  of  and  retain  the 
important  distinctions  that  individual  estimates  represent. 

Addressing  multiple  values  or  objectives  in  terrorism  risk  esti¬ 
mates  differs  from  combining  forecasts.  While  the  goal  of  combining 
forecasts  is  to  develop  an  accurate  estimate,  the  goal  of  considering 
multiple  objectives  is  to  reflect  appropriately  the  range  of  values  held 
by  stakeholders.  Literature  on  multiobjective  decisionmaking  pro¬ 
vides  several  approaches  for  addressing  the  fact  that  terrorism  risk  can 
be  expressed  in  multiple  outcomes.  The  commonality  across  these 
methods  is  the  need  to  reflect  transparently  a  range  of  values  for  mul¬ 
tiple  objectives  in  the  decisionmaking  process. 


Simple  Versus  Complex  Risk  Indicators 

Despite  the  many  sources  of  uncertainty  surrounding  terrorism  risk, 
estimating  this  risk  is  necessary  for  informed  distribution  of  home¬ 
land  security  resources.  Approaches  that  have  been  used  in  policy 
analysis  for  estimating  terrorism  risk  are  bounded  by  two  generic 
categories:  simple  risk  indicators  and  event-based  models.  Each  ap¬ 
proach  reflects  the  components  of  terrorism  risk  (i.e.,  threat,  vulner- 
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ability,  and  consequences)  and  their  uncertainties  in  different  ways. 
As  examples  of  simple  indicators,  we  describe  how  population  and 
density-weighted  population  have  been  used  as  estimates  of  terrorism 
risk.  As  an  example  of  event-based  models,  we  describe  the  Risk 
Management  Solutions  (RMS)  Terrorism  Risk  Model.  These  two 
examples  allow  for  comparisons  that  illustrate  the  strengths  and 
weaknesses  of  each  approach. 

There  is  a  logical  link  between  population-based  indicators  and 
terrorism  risk.  An  argument  can  be  made  that  consequences  are  cor¬ 
related  with  population  and  threats  are  correlated  with  population 
density.  There  are  practical  benefits  for  using  simple  risk  indicators 
such  as  those  based  upon  population.  In  general,  the  metrics  for 
measuring  these  indicators  are  well  understood  and  measurable,  and 
data  is  widely  available.  The  main  limitation  of  these  simple  indica¬ 
tors  is  that  they  do  not  fully  reflect  the  interactions  of  threat,  vulner¬ 
ability,  and  consequences.  As  a  result,  there  is  little  consensus  and  no 
validated  framework  for  deciding  how  to  use  several  simple  indicators 
to  create  a  single  risk  estimator. 

Event-based  models  are  built  upon  relatively  detailed  analysis  of 
consequences  from  specific  attack  scenarios.  These  models  include 
sensitivity  analysis  for  important  parameters  that  affect  consequences. 
They  may  include  components  to  model  multiple  types  of  events  and 
multiple  targets.  They  may  also  include  modules  that  translate  expert 
judgments  of  likelihood  or  consequences.  The  strength  of  event-based 
models  lie  in  the  greater  fidelity  they  enable  in  analysis.  The  weakness 
is  that,  to  obtain  this  detail,  analysts  must  estimate  many  uncertain 
parameters.  One  example  of  an  event-based  model  is  the  RMS  Ter¬ 
rorism  Risk  Model.  The  RMS  model,  discussed  in  more  detail  later  in 
this  monograph,  was  developed  as  a  tool  for  the  insurance  and  rein¬ 
surance  industries  to  assess  risks  of  macroterrorism.1 


1  RMS  defines  macroterrorism  as  attacks  capable  of  causing  (1)  economic  losses  in  excess  of 
$1  billion,  or  (2)  more  than  100  fatalities  or  500  injuries,  or  (3)  massively  symbolic  damage. 
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Estimating  Terrorism  Risk 

To  demonstrate  an  approach  for  estimating  terrorism  risk  to  inform 
resource  allocations,  we  calculated  a  single  estimate  of  U.S.  cities’  risk 
shares  based  on  multiple  perspectives  of  terrorism  risk  obtained  from 
the  RMS  Terrorism  Risk  Model.  By  considering  three  perspectives  on 
threat  (the  RMS  standard  and  enhanced  and  reduced  threat  out¬ 
looks),  the  RMS  results  provide  three  estimates  of  terrorism  risk  for 
the  urban  areas  that  received  UASI  funding  in  fiscal  year  2004.  Using 
these  three  sets  of  expected  fatalities,  we  calculated  an  aggregated  risk 
estimator  by  minimizing  the  sum  of  the  squared  underestimation  er¬ 
ror  across  all  urban  areas  and  risk  estimates.  The  assumptions  and 
limitations  inherent  to  this  approach  are  presented  in  the  monograph. 


Strengths  and  Weaknesses  of  Different  Risk  Indicators 

We  compared  the  proposed  aggregated  risk  estimator  to  simple 
population-based  indicators,  looking  at  both  how  the  distribution  of 
risk  changes  for  each  and  the  propensity  for  each  to  underestimate  a 
city’s  risk  share  given  the  uncertainty  that  surrounds  terrorism  risk. 

The  aggregated  risk  estimator  concentrated  most  of  the  expected 
terrorism  losses  in  relatively  few  cities  compared  to  population-based 
indicators.  In  addition,  the  aggregated  estimator  resulted  in  the  low¬ 
est  underestimates  of  risk  aggregated  across  all  urban  areas.  Density- 
weighted  population  performed  better  than  population  alone  and 
was,  in  fact,  quite  a  bit  closer  in  performance  to  that  of  the  aggregated 
indicator  than  to  that  of  the  simple  population  indicator. 

Because  the  density-weighted  population  indicator  performs 
well  and  is  easier  to  derive  than  the  event-based  indicator,  it  might  be 
of  utility  for  some  purposes,  e.g.,  in  risk-based  allocation  of  resources 
for  strategic  purposes  over  long  time  intervals,  during  which  relative 
risk  across  urban  areas  is  not  expected  to  change  much. 

Density-weighted  population,  however,  does  not  allow  deci¬ 
sionmakers  to  see  how  changes  in  threat  or  vulnerability  information 
affect  terrorism  risk.  For  example,  when  making  operational  resource 


Summary  xxi 


allocations  or  evaluating  the  effectiveness  of  preparedness  programs, 
decisionmakers  need  to  understand  how  specific  countermeasures  re¬ 
duce  or  change  the  profile  of  terrorism  risk.  Similarly,  a  crude  indica¬ 
tor  like  density-weighted  population  would  offer  no  guidance  about 
how  city  risk  estimates  might  change  with,  for  instance,  new  intelli¬ 
gence  about  terrorist  targeting  or  capabilities  of  using  weapons  of 
mass  destruction  (WMD).  For  these  purposes,  more  detailed  event- 
based  models  of  terrorism  risk  are  essential. 

In  this  study,  a  single  event-based  estimate  was  shown  to  be  ro¬ 
bust  across  uncertainties  about  the  likelihood  of  WMD  attacks,  un¬ 
certainties  about  which  consequences  ought  to  be  prioritized  in  con¬ 
siderations  of  city  risk,  and  uncertainties  about  the  expected 
magnitude  of  risks  each  city  might  face.  For  example,  an  important 
observation  is  that  the  risk  profile  of  the  urban  areas  examined  did 
not  change  significantly  with  the  variability  of  threats  from  weapons 
of  mass  destruction.  This  is  clearly  a  function  of  the  models  used  in 
this  study  and  how  they  were  parameterized.  While  the  primary  focus 
of  this  study  was  not  to  estimate  precisely  terrorism  risk  in  the  United 
States,  this  observation  raises  questions  such  as  whether  risk  is  a  char¬ 
acteristic  of  a  region’s  infrastructure  or  population  that  is  relatively 
stable  across  different  threats.  If  so,  this  would  be  an  important  ob¬ 
servation  when  it  comes  to  policy  and  resource  decisions. 


Recommendations 

Our  framework  for  defining  terrorism  risk  and  the  analysis  we 
present  here  lead  us  to  five  recommendations  for  improving  the  allo¬ 
cation  of  homeland  security  resources: 

1.  DHS  should  consistently  define  terrorism  risk  in  terms  of  ex¬ 
pected  annual  consequences.  Calculating  expected  annual  conse¬ 
quences  requires  accounting  for  threat,  vulnerability,  and  conse¬ 
quences.  Defining  terrorism  risk  in  these  terms  facilitates  the 
incorporation  of  risk  reduction  as  the  goal  of  homeland  security  pro¬ 
grams. 
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2.  DHS  should  seek  robust  risk  estimators  that  account  for  un¬ 
certainty  about  terrorism  risk  and  variance  in  citizen  values.  Given 
the  tremendous  uncertainties  surrounding  terrorism  risk  assessment, 
it  is  prudent  to  plan  for  the  range  of  plausible  futures  that  may  play 
out.  Several  approaches  are  available  for  generating  estimates  of  city 
risk  shares  that  offer  robust  characterizations  of  risk  across  multiple 
uncertainties  and  perspectives  on  relative  values  of  different  conse¬ 
quences.  Our  approach  to  this  problem  ensures  that  underestimation 
error  is  minimized. 

3.  DHS  should  develop  event-based  models  of  terrorism  risk, 
like  that  demonstrated  in  this  monograph.  Measuring  and  tracking 
levels  of  terrorism  risk  is  an  important  component  of  homeland  secu¬ 
rity  policy.  These  data  provide  insight  into  how  current  programs  are 
reducing  risk  and  when  and  where  new  terrorist  threats  may  be 
emerging.  Only  event-based  models  of  terrorism  risk  provide  insight 
into  how  changes  in  assumptions  or  actual  levels  of  threat,  vulner¬ 
ability,  and  consequences  affect  risk  levels. 

4.  Until  reliable  event-based  models  are  constructed,  density- 
weighted  population  should  be  preferred  over  population  as  a  sim¬ 
ple  risk  indicator.  Density-weighted  population  is  reasonably  corre¬ 
lated  with  the  distribution  of  terrorism  risk  across  the  United  States, 
as  estimated  by  event-based  models  like  the  RMS  Terrorism  Risk 
Model.  To  support  strategic  policy  decisions  when  the  effects  of  new 
countermeasures  or  recent  intelligence  are  not  in  question,  density- 
weighted  population  is  a  useful  indicator  in  lieu  of  event-based  mod¬ 
els.  In  contrast,  our  results  suggest  that  population  offers  a  remarka¬ 
bly  weak  indicator  of  risk,  not  much  superior  to  estimating  risk  shares 
at  random. 

5.  DHS  should  fund  research  to  bridge  the  gap  between  terror¬ 
ism  risk  assessment  and  resource  allocation  policies  that  are  cost  ef¬ 
fective.  We  do  not  here  seek  to  understand  how  UASI  allocation 
amounts  may  reduce  risk.  Until  that  relationship  is  understood,  re¬ 
source  allocation  decisions  will  not  be  optimized  for  reducing  casual¬ 
ties  and  property  loss.  To  these  ends,  DHS  should  evaluate  the  per¬ 
formance  of  the  formula  used  to  assign  UASI  grants  using  the 
approach  presented  in  this  study. 
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CBRN 

chemical,  biological,  radiological,  or  nuclear 

CTRMP 

Center  for  Terrorism  Risk  Management  Policy 

DHS 

Department  of  Homeland  Security 

E[X] 

expected  value  of  X 

LNG 

liquefied  natural  gas 

P(X) 

probability  that  event  X  will  occur 

P(X|Y) 

probability  that  event  X  will  occur,  given  that  event  Y  al¬ 
ready  has  occurred 

RMS 

Risk  Management  Solutions 

TNT 

trinitrotoluene 

TRIA 

Terrorism  Risk  Insurance  Act  of  2002 

UASI 

Urban  Areas  Security  Initiative 

U.S. 

United  States 

WMD 

weapons  of  mass  destruction 
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CHAPTER  ONE 

Introduction 


The  Urban  Areas  Security  Initiative  (UASI)  is  a  Department  of 
Homeland  Security  (DHS)  grant  program  designed  to  enhance  secu¬ 
rity  and  overall  preparedness  to  prevent,  respond  to,  and  recover  from 
acts  of  terrorism.  These  goals  are  accomplished  by  providing  financial 
assistance  to  address  the  unique  planning,  equipment,  training,  and 
exercise  needs  of  large  urban  areas  (DHS,  2004). 

In  fiscal  year  2004,  UASI  provided  $675  million  to  50  urban  ar¬ 
eas  perceived  to  be  at  highest  risk  from  terrorist  attacks.  These  funds 
were  allocated  using  a  formula  that  accounted  for  several  indicators  of 
the  terrorism  risk  to  which  each  urban  area  might  be  exposed. 
Though  precise  details  of  the  formula  are  not  publicly  available,  it 
reportedly  calculates  each  urban  area’s  share  of  total  terrorism  risk 
based  on  city  population,  infrastructure,  and  threat  information, 
giving  indicators  for  each  factor  an  importance  weight  of  nine,  six, 
and  three,  respectively.  Despite  this  effort  to  allocate  homeland  secu¬ 
rity  resources  based  on  the  relative  risks  to  which  each  urban  area  is 
exposed,  the  Department  of  Homeland  Security  has  frequently  been 
criticized  for  inadequately  calculating  risk  and  therefore  for  failing  to 
distribute  resources  in  proportion  to  urban  areas’  shares  of  total  ter¬ 
rorism  risk  (U.S.  House  of  Representatives,  2003). 

Debates  about  the  proper  allocation  of  resources  have  suffered 
from  several  problems.  For  instance,  currently,  there  is  no  shared  and 
precise  definition  of  terrorism  risk,  so  stakeholders  in  the  debate  are 
often  referring  to  different  concepts  of  risk.  Even  if  a  precise  defini¬ 
tion  were  widely  used,  there  are  no  standard  methods  for  estimating 
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and  monitoring  changes  in  the  level  and  nature  of  terrorism  risks.  In¬ 
stead,  various  indicators  of  risk  have  been  used  (for  instance  in  the 
UASI  formula)  or  proposed  (e.g.,  Canada,  2003),  which  are  pre¬ 
sumed  to  correspond  in  some  way  with  true  terrorism  risk.  To  our 
knowledge,  however,  no  systematic  frameworks  for  selecting  these 
indicators  or  aggregating  them  into  a  unitary  measure  of  risk  are  yet 
available.  Moreover,  terrorism  risk  changes  over  time  as  terrorist  mo¬ 
tives,  capabilities,  and  targets  change  and  adapt  to  risk-mitigation  ef¬ 
forts.  These  facts  defy  the  efficacy  of  any  simplistic  model  that  at¬ 
tempts  to  enumerate  a  single  index  as  a  measure  of  risk.  Measuring 
terrorism  risk  must  always  reflect  uncertainties  in  estimates  of  the 
relative  risks  faced  by  different  cities. 


Risk  Assessment  Versus  Resource  Allocation 

Ultimately,  efficient  allocation  of  homeland  security  resources  would 
be  determined  based  upon  assessment  of  the  cost  effectiveness  of  al¬ 
ternative  risk-reduction  opportunities.  This  requires  understanding 
the  cost  effectiveness  of  different  types  and  amounts  of  investment. 
As  a  hypothetical  example,  even  if  terrorism  risks  were  greater  in  New 
York  City  than  in  Des  Moines,  Iowa,  allocating  resources  according 
to  proportion  of  risk  would  not  be  optimal  if  available  countermea¬ 
sures  are  more  cost  effective  in  Des  Moines.1  For  example,  terrorists 
could  respond  strategically  to  countermeasures  in  New  York  City  and 
target  less-protected  areas,  or  the  marginal  effectiveness  of  resources 
spent  in  New  York  City  may  decrease  with  continuing  investment. 
Neither  the  methods  nor  the  data  are  available  to  answer  questions 
about  the  effectiveness  of  available  risk-reduction  alternatives  or  to 
determine  reasonable  minimum  standards  for  community  prepared¬ 
ness.  Until  these  questions  are  answered,  allocating  homeland  security 
resources  based  on  risk  is  the  next  best  approach  since  areas  at  higher 
risk  are  likely  to  have  more  and  larger  opportunities  for  risk  reduction 


1  For  discussions  of  how  terrorist  strategy  affects  resource  allocation  decisionmaking,  see 
Woo  (2002a,  2002b)  and  Lakdawalla  andZanjani  (2004). 
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than  are  areas  at  lower  risk.  That  is,  resources  would  be  allocated 
roughly  proportionally  to  the  distribution  of  risk  across  areas  receiv¬ 
ing  funding. 

There  are  several  other  reasons  why  it  is  still  important  for  deci¬ 
sionmakers  to  understand  the  levels  and  distribution  of  terrorism  risk. 
First,  because  assessing  risk  and  risk  reduction  is  a  critical  first  step  in 
assessing  cost  effectiveness  of  counterterrorism  efforts,  methods  de¬ 
veloped  to  support  terrorism  risk  assessment  will  also  support  analysis 
of  resource  allocation.  Further,  even  when  large  risks  are  not  miti¬ 
gated  by  current  efforts,  identifying  them  can  help  direct  intelligence 
gathering,  research,  and  future  counterterrorism  efforts.  Finally,  fol¬ 
lowing  changes  in  the  levels  and  patterns  of  terrorism  risk  over  time 
provides  insights  into  the  effectiveness  of  current  efforts  and  the 
emergence  of  new  risks. 


Scope  and  Limitations 

In  this  monograph,  we  propose  a  specific  definition  of  terrorism  risk 
that  can  be  operationalized  for  practical  problems  facing  DFiS  and 
develop  a  method  of  constructing  a  single  measure  of  risk  that  ac¬ 
counts  for  uncertainties  in  risk  measurement.  We  then  propose  and 
demonstrate  a  framework  for  evaluating  this  measure,  along  with  al¬ 
ternative  measures  of  risk,  to  understand  resulting  errors  given  uncer¬ 
tainties  in  their  measurement.  Finally,  we  offer  recommendations  on 
future  efforts  to  calculate  the  shares  of  total  terrorism  risk  to  which 
different  areas  are  exposed. 

While  the  discussions  in  this  monograph  focus  on  a  specific 
program,  the  UASI  grants,  the  problems  discussed  previously  are 
common  to  a  number  of  risk  estimation  problems  in  homeland  secu¬ 
rity.  Thus,  the  problem  is  a  general  one  of  decisionmaking  under  un¬ 
certainty,  and  the  solutions  presented  here  are  also  generalizable  to 
similar  decision  contexts. 

This  monograph  does  not  address  all  problems  identified  previ¬ 
ously.  This  treatment  of  risk  estimation  does  not  fully  inform  specifi¬ 
cation  of  a  formula  for  risk-based  allocation  of  homeland  security  re- 
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sources.  As  noted  previously,  before  such  a  formula  can  be  con¬ 
structed,  additional  research  is  needed  to  understand  the  relationship 
between  resource  allocation  and  risk  reduction. 

The  scope  of  this  project  is  further  limited  to  the  direct  conse¬ 
quences  associated  with  terrorism  threats.  Thus,  we  do  not  include  in 
our  estimates  of  terrorism  risk  the  secondary  and  higher  order  eco¬ 
nomic  or  other  losses  that  result  from  an  attack  on  a  given  location. 
These  effects  are  important  and  may  well  constitute  the  major  por¬ 
tion  of  the  risk  but  can  be  handled  by  the  methods  we  develop  here 
given  additional  resources.  Such  analysis  would  extend  this  current 
work  to  further  improve  the  evaluation  of  the  relative  risks  to  which 
different  urban  areas  are  exposed  and  thus  would  help  to  improve  the 
allocation  of  homeland  security  resources. 


Overview  of  This  Monograph 

The  remainder  of  this  monograph  is  organized  as  follows.  Chapter 
Two  defines  terrorism  risk  and  the  factors  it  comprises.  Chapter 
Three  discusses  the  sources  of  uncertainty  that  must  be  addressed 
when  incorporating  terrorism  risk  assessment  into  policy  decision¬ 
making  and  provides  guidance  on  how  each  form  of  uncertainty  can 
be  addressed.  In  Chapter  Four,  we  discuss  alternative  approaches  to 
estimating  levels  of  terrorism  risk  across  UASI-funded  urban  areas 
and  propose  a  method  for  combining  diverse  risk  estimates  into  a  sin¬ 
gle  estimate  of  each  urban  area’s  share  of  total  terrorism  risk.  In 
Chapter  Five,  we  develop  a  framework  for  evaluating  the  performance 
of  different  risk  estimators  given  uncertainties  about  the  distribution 
of  true  terrorism  risk  and  the  diversity  of  types  of  consequences  that 
must  be  considered.  Finally,  we  discuss  the  performance  of  the  alter¬ 
native  risk  estimators  and  the  implications  our  findings  have  for 
homeland  security  policy. 


CHAPTER  TWO 

Terrorism  Risk  and  Its  Components 


Differing  notions  of  terrorism  risk  frequently  fuel  disagreements 
about  the  relative  risks  to  which  different  regions  or  cities  are  ex¬ 
posed.  Some  arguments  implicitly  link  risk  to  terrorism  threats.  If,  for 
example,  one  city  were  known  through  gathered  intelligence  or  past 
history  to  be  the  preferred  target  for  terrorists,  this  view  would  sup¬ 
port  a  claim  that  this  city  has  a  high  level  of  terrorism  risk.  Alterna¬ 
tively,  others  argue  that  risk  is  more  closely  associated  with  infrastruc¬ 
ture  vulnerabilities  within  a  region  because  these  represent  logical 
targets  for  terrorism.  Thus,  for  example,  even  if  we  do  not  know  of  a 
threat  to  a  nuclear  power  plant,  reason  and  prudence  argue  that  we 
should  include  that  facility  in  considering  a  region’s  risk.  Finally,  dis¬ 
cussions  of  risk  occasionally  emphasize  the  possible  consequences  of 
terrorist  attacks  in  evaluating  risk.  Thus,  if  two  cities  have  similar 
chemical  storage  facilities,  but  one  has  the  facility  located  close  to  its 
population  center,  a  persuasive  argument  can  be  made  that  the 
nearer-to-population  city’s  chemical  facility  presents  a  greater  risk 
than  the  other  city’s. 

Clearly,  strong  arguments  can  be  made  that  threats,  vulnerabili¬ 
ties,  and  consequences  play  a  significant  part  in  the  overall  risk  to 
which  a  city  is  exposed.  What  has  been  less  clear  is  how  these  three 
components  are  related.  In  this  chapter,  we  offer  a  definition  of  risk 
that  links  them.  We  also  distinguish  between  definitions  for  threat, 
vulnerability,  consequences,  and  risks  and  the  measures  that  can  be 
used  to  assess  and  track  each. 
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Threat 

People  or  organizations  represent  a  terrorist  threat  when  they  have  the 
intent  and  capability  to  impose  damage  to  a  target.  Note  that  neither 
intentions  without  capabilities  nor  capabilities  without  intentions 
pose  a  threat.  Threat  only  exists  when  both  are  manifested  together  in 
a  person  or  organization.  Allocating  homeland  security  resources  to 
protect  critical  infrastructure  or  cities  requires  measuring  the  threats 
posed  to  specific  targets  or  from  specific  types  of  attack.  When  the 
scope  of  threat  is  defined  in  terms  of  a  specific  set  of  targets,  a  specific 
set  of  attack  types,  and  a  specific  time  period,  probability  can  be  used 
as  a  measure  of  the  likelihood  that  an  attack  will  occur.  Thus,  we  de¬ 
fine  a  measure  of  threat  as  follows: 

Measure  (Threat):  The  probability  that  a  specific  target  is  at¬ 
tacked  in  a  specific  way  during  a  specified  time  period,  or 


Threat  =  P( attack  occurs) 

This  measure  of  terrorist  threat  emphasizes  a  specific  type  of  attack 
on  specific  targets.  Radiological  attack  represents  a  different  threat  to 
a  specific  target  than  nuclear  attack.  Attacks  on  stadiums  represent 
different  threats  than  attacks  on  skyscrapers.  A  complete  description 
of  the  threats  to  which  a  target  is  exposed  would  require  considera¬ 
tion  of  every  mode  of  attack  separately.  In  practice,  however,  it  may 
suffice  to  focus  on  a  limited  number  of  attack  types  that  are  represen¬ 
tative  of  chemical,  biological,  radiological,  or  nuclear  (CBRN)  and 
explosive  attack  modes.  Similarly,  it  may  suffice  to  focus  on  a  limited 
number  of  target  types  or  groups  of  targets  in  a  region. 

This  measure  of  threat  is  specified  in  terms  of  attack  types  and 
targets.  The  intelligence  community  more  customarily  considers 
threat  in  terms  of  groups  of  attackers  given  its  interest  identifying  and 
stopping  those  who  might  pose  a  threat.  An  attack-type  perspective  is 
more  useful  for  the  task  of  resource  allocation  because  the  decision 
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context  is  most  concerned  with  what  targets  are  threatened  than  with 
by  whom  and  why. 

Finally,  since  our  measure  for  threat  is  uncertain,  one  should 
keep  in  mind  that  it  can  also  be  represented  by  a  probability  distribu¬ 
tion,  not  a  point  estimate.  These  definitions  are  consistent  with 
methods  and  terminology  proposed  through  applications  of  engi¬ 
neering  risk  analysis  to  terrorism  risk  assessment  (Ayyub,  2005;  Pate- 
Cornell,  2005;  von  Winterfeldt  and  Rosoff,  2005). 


Vulnerability 

Clearly,  not  all  threats  of  the  same  type  are  equally  important.  Fur¬ 
thermore,  the  threat  of  terrorism  is  dynamic  in  that  it  adapts  to  cur¬ 
rent  conditions  that  affect  the  likelihood  of  attack  success.  For  exam¬ 
ple,  even  if  a  typical  hotel  and  fortified  military  base  have  equal 
probability  of  being  subjected  to  a  car-bomb  attack,  the  attack  would 
be  more  likely  to  achieve  the  aim  of  causing  significant  damage  at  the 
less-secure  hotel.  Therefore,  we  also  need  a  precise  definition  of  vul¬ 
nerability  that  captures  information  about  the  infrastructure  in  which 
we  are  interested. 

Paraphrasing  Haimes,  vulnerability  is  the  manifestation  of  the  in¬ 
herent  states  of  the  system  (e.g.,  physical,  technical,  organizational,  cul¬ 
tural)  that  can  result  in  damage  if  attacked  by  an  adversary. 1  Referring 
again  to  the  domain  of  engineering  risk  analysis,  where  threat  can  be 
thought  of  as  being  a  load  or  force  acting  on  a  system,  vulnerability 
can  be  thought  of  as  being  the  capacity  of  a  system  to  respond  to  this 
threat  (Pate-Corn ell,  2005).  To  use  this  definition  for  measurement, 
we  must  be  more  specific  and  ask,  “vulnerable  to  what?”  Probability 
can  be  used  as  a  measure  of  the  likelihood  that  vulnerability  will  lead 
to  damage  when  attacks  occur. 


1  Haimes  (2004,  p.  699).  We  note  that  most  of  this  italicized  phrase  is  verbatim  from  this 
source,  but  we  have  changed  the  definition  slightly  so  as  not  to  imply  that  an  attacker  needs 
to  exploit  a  vulnerability  knowingly — that  is,  a  target  can  be  vulnerable  without  the  vulner¬ 
ability  being  recognized  by  an  attacker. 
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Measure  (Vulnerability):  The  probability  that  damages  (where 
damages  may  involve  fatalities,  injuries,  property  damage,  or  other  conse¬ 
quences )  occur,  given  a  specific  attack  type,  at  a  specific  time,  on  a  given 
target,  or 


Vulnerability  =  P{ attack  results  in  damage  |  attack  occurs) 

In  other  words,  a  target’s  vulnerability  can  be  articulated  as  the  prob¬ 
ability  that  an  attack  of  a  given  type  will  be  successful  once  it  has 
been  launched  and,  as  articulated,  measures  vulnerability  to  specific 
types  of  damages  only  (i.e.,  there  would  be  separate  vulnerability  as¬ 
sessments  for  deaths,  injuries,  and  property  damage). 

Note  that  for  the  measure  specified  above,  magnitude  of  the 
damage  is  not  part  of  the  definition  of  vulnerability.  This  measure 
assumes  a  simplified  representation  of  vulnerability  in  which  there  is 
either  a  successful  attack  with  damage  or  no  success  with  no  damage. 
As  a  result,  we  define  “success”  in  terms  of  whether  or  not  damage, 
having  a  distribution  of  magnitude,  is  inflicted  by  the  attack.  Conse¬ 
quence  measurement  is  discussed  below.  A  more  general  model  (used 
in  many  military  analyses)  is  that  there  is  a  range  of  damage  levels, 
each  associated  with  its  own  probability.  This  is  simply  a  more  dis¬ 
crete  representation  of  damage  and  defense  mechanisms. 


Consequences 

We  define  “consequence”  as  the  magnitude  and  type  of  damage  re¬ 
sulting  from  successful  terrorist  attacks.  To  define  a  measure  of  con¬ 
sequence,  specificity  is  again  required.  In  this  case,  specificity  requires 
treatment  of  two  important  considerations:  how  consequences  are 
measured  and  how  uncertainty  is  addressed.  Formally,  we  state  this  as 
follows: 
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Measure  (Consequence):  The  expected  magnitude  of  damage 
(e.g.,  deaths,  injuries,  or  property  damage),  given  a  specific  attack  type,  at 
a  specific  time,  that  results  in  damage  to  a  specific  target  or, 


Consequence  =  E  (damage  |  attack  occurs  and  results  in  damage) 

Consequences  can  be  expressed  in  terms  of  fatalities,  injuries,  eco¬ 
nomic  losses,  or  other  types  of  damage.  Other  aspects  of  conse¬ 
quences  can  also  be  considered  using  the  approach  we  outline  here 
and  this  definition.  For  example,  the  damage  or  destruction  of  critical 
infrastructures  that  cause  injury,  loss  of  life,  and  economic  damage 
outside  the  area  of  immediate  attack  are  important.  They  may  in  fact 
dominate  the  results  of  an  analysis  if  the  impact  on  society  as  a  whole 
is  considered  rather  than  the  impact  on  the  target  and  its  occupants 
and  owners.2  In  this  monograph,  however,  we  limit  our  focus  to  mor¬ 
tality,  morbidity,  and  economic  loss  at  the  point  of  attack  in  order  to 
illustrate  an  approach  to  risk  estimation  in  a  manner  that  is  transpar¬ 
ent  yet  relevant  to  real-world  policy  decisions. 

As  discussed  in  detail  in  Chapter  Three,  consequences  are  de¬ 
termined  by  many  uncertain  factors,  such  as  wind  speed  or  relative 
humidity  (which  could  be  important  factors  in  a  chemical  or  biologi¬ 
cal  attack,  for  example).  These  uncertainties  can  be  addressed  by  con¬ 
sidering  a  full  distribution  for  potential  consequences  or  specific 
points  along  this  distribution.  Haimes  (2004)  notes  that  risk  assess¬ 
ment  of  rare  and  extreme  events  requires  special  consideration  of  the 
tails  of  these  distributions,  and  that  the  expected  value  often  misrep¬ 
resents  true  risk.  Conversely,  estimates  of  the  tail  of  the  distribution 
will  be  very  dependent  upon  assumptions  when  considering  events 
like  terrorism  where  there  is  great  uncertainty  about  events  and  lim¬ 
ited  historical  information.  For  this  reason,  and  to  simplify,  our  con¬ 
tinued  discussion  of  consequences  considers  the  expected  value  of  the 
distribution  of  damage. 


2  See  Rinaldi,  Peerenboom,  and  Kelly  (2001)  for  a  comprehensive  discussion  of  these  topics. 
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Risk  as  a  Function  of  Threat,  Vulnerability,  and 
Consequences 

Risk  is  the  anticipated  consequences  over  some  period  of  time  to  a 
defined  set  of  targets,  resulting  from  a  defined  set  of  threats.  For  a 
specific  threat,  target,  and  type  of  consequence,  risk  can  be  measured 
as  follows: 

Measure  (Terrorism  Risk):  The  expected  consequence  of  an  exis¬ 
tent  threat,  which  for  a  given  target,  attack  mode,  and  damage  type  can 
be  expressed  as 


Risk  =  P( attack  occurs) 

*  /’(attack  results  in  damage  |  attack  occurs) 

*  E  (damage  |  attack  occurs  and  results  in  damage) 

=  Threat  *  Vulnerability  *  Consequence 

In  other  words,  terrorism  risk  represents  the  expected  consequences 
of  attacks  taking  into  account  the  likelihood  that  attacks  occur  and 
that  they  are  successful  if  attempted.  In  probabilistic  terms,  risk  from 
an  attack  of  a  certain  type  is  the  unconditional  expected  value  of 
damages  of  a  certain  type. 

There  are  two  advantages  of  using  this  formulation  of  terrorism 
risk.  First,  it  provides  an  approach  for  comparing  and  aggregating  ter¬ 
rorism  risk.  With  this  definition,  it  is  possible  to  compare  risks  of  a 
specific  type  across  diverse  targets  such  as  airports  and  electrical  sub¬ 
stations.  For  example,  the  injury  risk  from  an  explosives  attack  could 
be  expressed  for  both  as  the  expected  annual  injuries  resulting  from 
such  attacks  against  each  target.  Estimating  overall  terrorism  risk  re¬ 
quires  further  analysis  that  considers  all  threat  types  and  targets.  If 
risks  were  independent,  expected  damages  of  a  specific  type  could  be 
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aggregated  by  summing  across  threat  types  and  target  types.3  How¬ 
ever,  dependencies  likely  exist  between  risks.  For  instance,  a  successful 
nuclear  attack  in  a  city  could  dramatically  change  the  expected  risks 
for  targets  in  the  damage  footprint  of  the  explosion. 

Second,  this  definition  of  risk  provides  a  clear  mapping  between 
risk  and  approaches  to  managing  or  reducing  risk.  Intelligence  and 
active  defense  involving  “taking  the  fight  to  the  enemy”  represent  an 
approach  to  risk  management  that  focuses  specifically  on  threats. 
Managing  risk  through  vulnerability  requires  increasing  surveillance 
and  detection,  hardening  targets,  or  other  capabilities  that  might  re¬ 
duce  the  success  of  attempted  attacks.  Finally,  risk  can  be  managed 
through  consequences  by  increasing  preparedness  and  response  that 
reduces  the  effects  of  damage  through  mitigation  or  compensation. 


3  Damage  of  different  types  (i.e.,  casualties  versus  economic  damages)  should  be  treated  us¬ 
ing  approaches  of  multiobjective  decisionmaking,  not  simple  aggregation.  This  is  discussed 
further  in  Chapter  Three. 


CHAPTER  THREE 

Accounting  for  Uncertainty  and  Values  in 
Terrorism  Risk  Assessment 


The  definitions  and  measures  presented  in  Chapter  Two  provide  a 
simplified  perspective  on  threat,  vulnerability,  consequences,  and  risk 
that  is  useful  for  thinking  about  homeland  security  and  preparedness. 
The  reality  that  threat,  vulnerability,  and  consequences  are  all  subject 
to  tremendous  uncertainties  makes  estimating  each  a  challenging  task. 
To  facilitate  risk  estimation,  it  is  important  to  understand  the  sources 
of  these  uncertainties  that  affect  terrorism  risk. 

There  are  two  important  sources  of  uncertainty  in  estimating 
terrorism  risk.  The  first  includes  variability  and  error  in  estimates  of 
threats,  vulnerabilities,  and  consequences.  For  example,  exact  knowl¬ 
edge  of  the  threat  would  require  comprehensive  intelligence  on  the 
plans  and  capabilities  of  all  terrorist  groups.  Since  this  level  of  preci¬ 
sion  is  not  feasible,  expert  judgments  must  be  substituted  for  fact, 
resulting  in  parameter  estimates  for  threats  that  are  subject  to  uncer¬ 
tainty  or  frank  disagreements. 

The  second  source  of  uncertainty  concerns  how  we  should  value 
different  types  of  consequences.  This  is  a  fundamental  problem  un¬ 
derlying  homeland  security  decisions  that  inevitably  share  burdens  of 
cost  and  risk  among  different  parts  of  U.S.  society.  For  instance,  if  a 
city  has  a  small  property-loss  risk,  but  a  large  fatality  risk,  you  might 
conclude  that  it  has  medium  overall  terrorism  risk  by  valuing  the  im¬ 
portance  of  each  type  of  consequence  as  roughly  equivalent.  In  fact, 
any  strategy  for  judging  the  relative  importance  of  different  types  of 
consequence  represents  an  attempt  to  estimate  the  value  that  we  (so¬ 
ciety  or  the  United  States)  should  or  does  place  on  each  consequence. 
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Because  this  requires  value  judgments — and  potentially  contentious 
ones — it  must  ultimately  be  discussed  by  the  public  and  policymak¬ 
ers.  Part  of  an  informed  discussion  of  this  judgment  rests  on  an  un¬ 
derstandable  and  transparent  illustration  of  the  consequences  of  using 
alternative  values. 


Uncertainties  in  the  Estimation  of  Threat,  Vulnerability, 
and  Consequences 

At  any  moment,  we  may  assume  that  a  target  has  exact,  true  values 
for  threat,  vulnerability,  consequences  and  therefore  for  risk.  These 
values,  however,  cannot  be  directly  observed,  so  must  be  estimated. 
Estimation  introduces  uncertainty  and  error. 

Probably  the  greatest  source  of  uncertainty  derives  from  esti¬ 
mates  of  threat,  which  concerns  terrorists’  goals,  motives,  and  capa¬ 
bilities.  Our  sources  of  information  on  these  factors — chiefly  intelli¬ 
gence,  historical  analysis,  and  expert  judgment — support  only  crude 
estimates  of  the  probability  of  attacks  against  specific  targets  or  classes 
of  targets  (e.g.,  banks).  Experts  frequently  disagree  about  the  goals  of 
terrorist  groups  and  their  capabilities,  and  some  terror  groups  may 
exist  about  which  little  is  known.  Consequently,  assessments  of  ter¬ 
rorist  motivations  and  capabilities  may  systematically  under-  or  over¬ 
estimate  threats.  Given  this,  our  threat  estimates  must  be  treated  with 
suspicion. 

Vulnerability  estimates  may  be  subject  to  lower  levels  of  uncer¬ 
tainty.  Because  vulnerability  concerns  the  likelihood  that  an  attack  of 
a  specific  type  and  magnitude  will  be  successful  against  a  target,  it 
concerns  matters  that  can,  in  principle,  be  carefully  studied  and  for 
which  rough  estimates  may  be  reasonably  good,  e.g.,  the  methods  of 
engineering  risk  analysis  that  have  been  used  successfully  in  estimat¬ 
ing  risks  of  space  flight  and  operating  nuclear  reactors.  These  meth¬ 
ods  can  be  applied  to  protecting  targets  and  infrastructure  (Haimes, 
2004).  Despite  the  applicability  of  such  approaches,  imprecision  re¬ 
mains  in  the  estimation  of  a  target’s  vulnerability. 
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The  sources  of  uncertainty  in  consequences  concern  damage  as¬ 
sessments  that  depend  on  the  physical  situation  at  the  target  when  the 
attack  occurs.  For  example,  suppose  a  chemical  weapon  that  takes  the 
form  of  a  spray  of  fine  droplets  is  used  at  a  popular  oceanfront  recrea¬ 
tional  location  on  a  busy  weekend  afternoon  and  that  the  chemicals 
are  only  effective  while  they  remain  in  droplet  form — that  is,  they  are 
not  effective  after  they  evaporate.  If  the  atmospheric  conditions  are 
hot  and  dry,  and  the  wind  is  blowing  from  the  land  out  over  the  sea, 
then  the  droplets  might  evaporate  more  quickly  due  to  the  dry  heat 
and  will  blow  out  over  the  water  away  from  areas  where  many  people 
gather.  However,  if  the  weather  conditions  are  such  that  the  wind  is 
blowing  over  the  beach  on  a  humid  day,  the  damage  from  fatalities 
and  injuries  might  be  more  severe. 

This  example  illustrates  that  estimating  consequences  requires  a 
substantial  amount  of  work.  Fortunately,  this  work  is  often  in  the 
form  of  straightforward  engineering  and  statistical  problems,  and 
well-developed  models  exist  of  many  natural  disasters  that  are  directly 
applicable,  or  nearly  so.  Additionally,  the  military  and  other  govern¬ 
ment  agencies  have  long  studied  the  effects  of  weapons  on  people  and 
structures,  and  this,  too,  is  useful  for  estimating  consequences.  For 
example,  modeling  has  been  used  to  estimate  the  consequences  of  at¬ 
tacks  such  as  releases  of  hazardous  chemicals  or  biological  agents,  ra¬ 
diological  bombs,  conventional  explosions,  and  nuclear  detonations 
(Abt,  2003;  Bozette  et  ah,  2003;  RMS,  2003).  Other  approaches,  like 
the  Interoperability  Input-Output  model  (Haimes,  2004),  use  eco¬ 
nomic  data  to  understand  the  indirect  economic  consequences  of  at¬ 
tacks  resulting  from  interdependence  between  market  sectors.  Al¬ 
though  more  information  is  available  on  many  consequences,  the 
precision  with  which  that  information  may  be  applied  should  not  be 
overstated.  The  RMS  Terrorism  Risk  Model  described  in  Chapter 
Four  is  built  upon  thousands  of  estimates  from  these  types  of  detailed 
models.  Applying  detailed  event  models  requires  estimation  of  nu¬ 
merous  parameters,  a  process  that  itself  introduces  uncertainty. 
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Reflecting  Values  in  Terrorism  Risk  Management 

Models  of  terrorist  attacks  can  assess  impacts  in  terms  of  injuries  and 
fatalities,  property  loss,  economic  losses,  citizen  confidence  and  feel¬ 
ings  of  security,  or  myriad  other  potentially  relevant  outcomes.  Risk 
can  likewise  be  expressed  in  terms  of  any  one,  or  a  combination,  of 
these  consequences. 

The  emphasis  placed  on  each  type  of  consequence  in  the  evalua¬ 
tion  of  terrorism  risk  is  a  value  judgment.  While  all  types  of  risk 
could  mathematically  be  combined  into  a  single-dimensional  aggre¬ 
gate  risk,  any  such  aggregation  requires  making  value  judgments  on 
the  relative  importance  of  different  consequences.  This  multidimen¬ 
sional  nature  of  consequences  creates  difficult  decisions  for  policy¬ 
makers  who  must  weigh  the  relative  importance  of  different  types  of 
consequences  when  allocating  homeland  security  resources. 

As  an  example,  consider  two  regions.  The  first,  a  densely  popu¬ 
lated  business  district,  may  be  viewed  as  a  terrorist  target  for  being  a 
hub  of  economic  activity.  Terrorist  attacks  at  this  location  could  pro¬ 
duce  large  numbers  of  fatalities  and  vast  economic  losses.  The  second 
region,  an  industrial  park  where  petroleum  refining  and  transferring 
take  place,  could  also  be  considered  a  target.  For  this  case,  economic 
losses  may  be  equally  large;  however,  because  of  lower  population 
densities,  expected  fatalities  might  be  lower. 

The  priorities  given  to  these  two  regions  in  estimating  risk  are 
driven  by  values — in  other  words,  the  relative  weight  assigned  to  a 
particular  type  of  consequences.  Different  stakeholder  groups  will 
have  different  perspectives.  Some  may  believe  that  mitigating  risk  is 
exclusively  about  the  prevention  of  deaths  and  injuries,  and  thus  the 
business  district  bears  the  greatest  burden  of  risk.  Another  group  may 
value  minimizing  damage  to  the  domestic  economy  more,  and  conse¬ 
quently  believe  that  risk  is  divided  more  equally  between  the  two  re¬ 
gions.  In  our  system  of  allocating  homeland  security  resources,  deci¬ 
sionmakers  have  an  important  role  in  understanding  and  representing 
diverse  sets  of  values.  In  the  end,  the  approach  used  in  the  decision¬ 
making  process  should  allow  transparency,  so  that  citizens  can  effec- 
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tively  participate  in  risk-management  deliberations  (Stern  and  Fine- 
berg,  1996). 


Implications  of  Uncertainty  and  Values  for  Managing 
Terrorism  Risk 

The  uncertainties  described  earlier  ensure  there  will  be  disagreement 
on  how  to  model  terrorist  events  and  protection  strategies,  how  to 
specify  probability  distributions  to  represent  threat  or  vulnerability, 
and  how  to  value  diverse  measures  of  consequences.  Lempert,  Pop¬ 
per,  and  Bankes  (2003)  define  these  conditions  as  reflecting  a  state  of 
deep  uncertainty.  Deep  uncertainty  has  important  implications  for 
decisionmakers  charged  with  developing  policies  that  depend  on  ter¬ 
rorism  risk  assessments,  like  risk-based  allocation  formulae. 

Often  policy  analysis  relies  on  best  estimates  even  when  they 
have  a  low  probability  of  being  correct — and  a  high  probability  of 
being  wrong.  While  this  generates  a  very  precise  estimate  of  risk,  in 
the  end,  if  the  estimates  poorly  represent  what  actually  happens  in 
real  life,  the  precision  is  misplaced.  Lempert,  Popper,  and  Bankes 
(2003)  propose  an  alterative  approach  when  addressing  conditions  of 
deep  uncertainty.  The  approach  specifies  a  wide  range  of  future  sce¬ 
narios  that  could  unfold  and  then  challenges  decisionmakers  to 
choose  strategies  that  perform  well  across  a  large  number  of  these  pos¬ 
sible  futures,  rather  than  for  a  single  best  estimate  of  the  future.  This 
approach  is  consistent  with  methods  of  capabilities-based  planning 
that  the  Department  of  Defense  has  adopted  to  ensure  adaptiveness 
in  response  to  uncertainty  about  future  threats  (Davis,  1994,  2002). 

By  analogy,  if  we  believed  some  model  of  the  stock  market  was 
very  reliable  or  subject  to  low  rates  of  error,  then  we  would  allocate 
most  or  all  of  our  investment  in  those  stocks  it  predicts  to  rise.  That 
is,  we  would  base  our  policy  chiefly  on  the  model’s  best  estimates.  In 
contrast,  if  we  believed  the  same  estimates  were  subject  to  consider¬ 
able  error,  we  would  be  wise  to  hedge  our  investments  against  alterna¬ 
tive  possible  market  outcomes  within  the  range  of  plausible  futures 
suggested  by  the  uncertainty  around  our  model  estimates.  Similarly, 


18  Estimating  Terrorism  Risk 


in  this  chapter,  we  argue  that  uncertainty  in  terrorism  risk  estimates 
suggests  the  need  to  devise  means  of  hedging  our  homeland  security 
policies  against  a  range  of  distributions  of  risk  that  are  plausible  given 
what  we  know  about  uncertainties  in  our  risk-estimation  procedures. 

So,  rather  than  seek  an  optimal  method  for  estimating  risk,  we 
seek  a  method  that  leads  us  to  make  the  least  egregious  errors  in  deci¬ 
sionmaking  across  the  range  of  possible  scenarios  that  might  develop 
in  the  future.  In  other  words,  following  Lempert,  Popper,  and  Bankes 
(2003),  when  confronting  deep  uncertainty  we  seek  a  method  for  es¬ 
timating  risk  that  is  robust  because  it  has  the  lowest  expected  error 
when  evaluated  against  a  wide  range  of  possible  futures.  This  decision 
rule  is  conceptually  similar  to  Savage’s  minimax  principle  and  regret 
minimization  (Savage,  1951). 

One  approach  for  developing  an  estimate  with  these  properties 
would  be  first  to  define  multiple  sets  of  threats,  vulnerabilities,  and 
consequence  measures  and  use  them  as  the  basis  for  constructing  a 
single  risk  estimate.  Then,  using  these  multiple  estimates,  one  could 
develop  a  single  description  of  how  risk  is  distributed  that  balances 
across  multiple  perspectives  of  terrorism  risk.  This  estimate  would 
describe  the  distribution  of  aggregate  risk  and  would  potentially  be 
robust  across  a  wide  range  of  possible  futures  and  values.  That  is,  an 
estimate  may  be  designed  to  minimize  the  error  between  estimated 
risk  and  actual  risks  that  may  materialize  as  the  future  unfolds. 

Each  estimate  of  threats,  vulnerabilities,  and  consequences  repre¬ 
sents  a  different  view  of  what  is  valued  and  likely  to  take  place  in  the 
future.  These  views  could  be  results  from  multiple  parameterizations 
of  a  single  model,  results  from  multiple  models,  or  the  perspectives  of 
different  experts.  The  different  views  could  produce  estimates  of  the 
concentration  of  terrorism  risk  in  different  geographic  areas,  through 
different  modes  of  attack,  or  for  different  types  of  consequences.  For 
example,  in  each  view,  a  different  region  might  be  found  to  have  the 
greatest  share  of  terrorism  risk,  or  consequences  may  be  measured 
using  a  different  outcome. 

Generating  multiple  risk  estimates  provides  information  about 
the  bounds  of  understanding  about  terrorism  risk  and  how  different 
stakeholders  may  be  affected  based  on  where  they  live  or  what  out- 
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comes  they  value  most.  From  several  estimates,  one  can  ask  how  low 
or  high  terrorism  risk  may  be  in  a  specific  city,  what  a  best  estimate  of 
risk  is  given  the  range  of  estimates  available,  and  how  answers  to  these 
questions  differ  when  considering  different  types  of  outcomes.  The 
challenge  to  analysts  defining  a  single  picture  of  how  risk  is  distrib¬ 
uted  is  to  do  so  without  losing  significant  information.  This  requires 
specifying  how  to  deal  with  the  challenges  of  aggregation  given  both 
inherent  uncertainties  and  value  choices  discussed  previously. 

Aggregating  Risk  Estimates 

The  previous  discussion  suggests  that  uncertainty  in  terrorism  risk 
estimates  can  be  addressed  through  simple  aggregations  of  multiple 
perspectives  on  threat.  In  fact,  this  approach  has  proved  useful  in  an¬ 
other  field,  economic  forecasting.  Combining  economic  forecasts  is 
conceptually  similar  to  combining  estimates  of  terrorism  risk.  In  each 
case,  there  is  uncertainty  about  future  events  that  increases  as  one  fo¬ 
cuses  further  into  the  future.  For  both  terrorism  and  economic  fore¬ 
casting,  competing  models  exist  built  on  different  assumptions  about 
model  structure  and  parameters.  In  several  cases,  the  competing  mod¬ 
els  for  each  are  highly  correlated.  Clemen’s  (1989)  literature  review 
demonstrates  that  combining  economic  forecasts  can  improve  the 
accuracy  of  predictions  and  that  simple  aggregations  of  estimates, 
such  as  averaging,  can  perform  well  when  compared  to  more  complex 
methods  that  take  into  account  correlations  between  estimates  or 
judgments  of  forecast  quality.  Of  course,  the  hazard  of  aggregating 
forecasts  is  that  important  divergent  perspectives  might  be  lost.  For 
example,  Morgan  and  Keith  (1995)  demonstrated  that  expert  judg¬ 
ments  about  climate  change  reveal  tremendous  diversity  of  opinion. 
Thus,  it  is  important  to  achieve  two  goals  when  combining  forecasts: 
1)  achieving  the  potential  gains  in  accuracy  that  come  from  multiple 
forecasts  and  2)  make  note  of  and  retain  the  important  distinctions 
that  individual  forecasts  represent. 

Values  and  Multiobjective  Decisionmaking 

Addressing  multiple  values  or  objectives  in  terrorism  risk  estimates 
differs  from  combining  forecasts.  While  the  goal  of  combining  fore- 
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casts  is  to  develop  an  accurate  estimate,  the  goal  of  considering  mul¬ 
tiple  objectives  is  to  reflect  appropriately  the  range  of  values  held  by 
stakeholders.  For  example,  a  terrorism  risk  estimation  model  may 
yield  estimated  losses  in  terms  of  fatalities  and  economic  damage. 
Using  the  estimate  of  fatalities  is  not  more  accurate  or  inaccurate  than 
using  the  estimate  of  economic  damages.  Rather,  it  is  important  that 
decisionmakers  understand  and  consider  the  tradeoffs  across  conse¬ 
quence  types  (or  objectives)  when  selecting  among  policy  alternatives. 

Literature  on  multiobjective  decisionmaking  provides  several 
approaches  for  addressing  the  fact  that  terrorism  risk  can  be  expressed 
in  multiple  outcomes.  Methods  such  as  multiattribute  utility  analysis 
(Keeney  and  Raiffa,  1993)  and  multiobjective  value  models  (Keeney, 
1992)  emphasize  the  need  to  structure  decisions,  elicit  stakeholder 
preferences,  and  apply  axiomatic  rules  for  combining  outcomes. 
Tradeoff  analysis  (Hammond,  Keeney,  and  Raiffa,  1999;  Chankong 
and  Haimes,  1983)  reduces  the  burden  of  the  elicitation  process  by 
identifying  dominating  alternatives  and  eliminating  inferior  choices 
through  specification  of  equivalent  choices  across  objectives.  Hierar¬ 
chical  Holographic  Modeling  incorporates  multiple  objectives  by  cap¬ 
turing  and  representing  alternative  views  of  a  given  problem  for  deci¬ 
sionmakers  (Haimes,  2004).  The  commonality  across  these  methods 
is  the  need  to  reflect  transparently  a  range  of  values  for  multiple  ob¬ 
jectives  in  the  decisionmaking  process. 

Though  multiobjective  decisionmaking  is  an  essential  part  of  the 
policy  process,  the  remainder  of  this  monograph  provides  a  demon¬ 
stration  of  different  approaches  to  estimating  risk  and  combining  es¬ 
timates  without  attempting  to  characterize  stakeholders’  values  or 
preferences.  Instead,  we  only  discuss  risk  in  terms  of  expected  fatali¬ 
ties.  Thus,  to  the  extent  that  risks  are  distributed  differently  if  meas¬ 
ured  as  injuries  or  economic  consequences,  this  becomes  an  addi¬ 
tional  source  of  uncertainty  in  our  estimation  of  overall  risk.  A 
comprehensive  terrorism  risk  assessment  must  allow  decisionmakers 
to  understand  the  implications  of  different  value  judgments.  Thus, 
estimates  presented  in  subsequent  chapters  of  this  monograph  must 
be  interpreted  with  this  limitation  in  mind. 


CHAPTER  FOUR 

Two  Approaches  to  Estimating  Terrorism  Risk  in 
Urban  Areas 


Despite  the  many  sources  of  uncertainty  surrounding  terrorism  risk, 
estimating  this  risk  is  necessary  for  informed  distribution  of  home¬ 
land  security  resources.  This  chapter  describes  two  approaches  for  es¬ 
timating  terrorism  risk:  simple  risk  indicators  and  event-based  mod¬ 
els.  Each  approach  reflects  the  components  of  terrorism  risk  (i.e., 
threat,  vulnerability,  and  consequences)  and  their  uncertainties  in  dif¬ 
ferent  ways.  As  examples  of  simple  indicators,  we  describe  how 
population  and  density-weighted  population  have  been  used  as  esti¬ 
mates  of  terrorism  risk.  As  an  example  of  event-based  models,  we  de¬ 
scribe  the  RMS  Terrorism  Risk  Model.  These  two  examples  allow  for 
comparisons  that  illustrate  each  approach’s  strengths  and  weaknesses. 


Simple  Risk  Indicators:  Population-Based  Metrics 

Population  is  often  incorporated  into  simple  indicators  of  terrorism 
risk.  The  DHS  State  Homeland  Security  Grant  Program  uses  a  com¬ 
bination  of  equal  allocation  of  resources  and  use  of  population  as  a 
simple  proxy  for  terrorism  risk.  In  its  first  several  years  of  funding,  the 
program  allocated  its  first  cut  of  funding  so  that  each  state  received 
0.75  percent  of  the  total  resources.  The  second  cut  allocated  the  re¬ 
maining  funding  in  proportion  to  each  state’s  population  (Ransdell 
and  Boloorian,  2004).  Proposed  legislation  would  reduce  the  guaran¬ 
teed  amount  directed  to  each  state  and  require  that  the  remaining 
funds  are  allocated  with  consideration  of  threat.  The  UASI  grant  pro- 
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gram  allocation  formula,  mentioned  previously,  is  also  partially  based 
on  population. 

There  is  a  logical  link  between  population-based  indicators  and 
terrorism  risk.  An  argument  can  be  made  that  consequences  are  cor¬ 
related  with  population.  However,  terrorism  risks  to  a  population  of 
100,000  are  clearly  different  if  that  population  resides  in  a  dense  ur¬ 
ban  area  rather  than  if  it  is  spread  across  a  larger  rural  area  because  of 
the  higher  probability  of  many  high-profile  targets  and  more  people 
within  any  given  attack  footprint.  Density-weighted  population,  i.e., 
the  product  of  a  region’s  population  and  its  population  density,  offers 
one  of  many  possible  simple  risk  indicators  that  account  for  this  dif¬ 
ference.  Just  as  population  can  be  considered  correlated  with  conse¬ 
quences,  so  too  is  population  density  correlated  with  threat.  For  ex¬ 
ample,  a  terrorist  targeting  1,000  people  might  be  more  likely  to 
attack  a  group  when  they  are  all  within  the  same  city  block  than  if 
they  are  dispersed  across  the  country.  In  evaluating  alternative  ap¬ 
proaches  for  allocating  resources,  the  Congressional  Research  Service 
noted  that  population  and  density-weighted  population  are  correlated 
though  result  in  different  distributions  of  resources  (Canada  2003). 
There  are  practical  benefits  for  using  simple  risk  indicators  such  as 
those  based  upon  population.  In  general,  the  metrics  for  measuring 
these  indicators  are  well  understood,  measurable,  and  data  is  widely 
available.  For  example,  the  2000  U.S.  Census  is  a  very  credible  source 
for  data  on  the  population  and  population  density  for  the  UASI- 
funded  urban  areas.  Table  4.1  shows  this  data  along  with  the  funding 
each  area  received  in  FY2004. 
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Table  4.1 

Population,  Population  Density,  Density-Weighted  Population,  and  Grant 
Allocations  for  Urban  Areas  Receiving  UASI  Funding  in  Fiscal  Year  2004 


FY2004  UASI 


Urban  Areas 

Population3 

Population 
Density3  (per 
Square  Mile) 

Density- 

Weighted 

Population11 

Grant  Allo¬ 
cation' 

($  million) 

Albany-Schenectady-Troy 

875,583 

272 

237,926,588 

7 

Atlanta 

4,112,198 

672 

2,761,386,037 

11 

Baltimore 

2,552,994 

979 

2,498,144,264 

16 

Baton  Rouge 

602,894 

380 

229,154,762 

7 

Boston,  MA-NH 

3,406,829 

1,685 

5,740,709,241 

19 

Buffalo-Niagara  Falls 

1,170,111 

747 

873,657,856 

10 

Charlotte-Gastonia- 

1,499,293 

444 

665,682,378 

7 

Rock  Hill,  NC-SC 

Chicago 

8,272,768 

1,634 

13,519,096,414 

34 

Cincinnati,  OH-KY-IN 

1,646,395 

493 

811,141,960 

13 

Cleveland-Lorain-Elyria 

2,250,871 

832 

1,871,707,337 

10 

Columbus,  OH 

1,540,157 

490 

755,141,752 

9 

Dallas 

3,519,176 

569 

2,002,093,120 

12 

Denver 

2,109,282 

561 

1,183,064,989 

9 

Detroit 

4,441,551 

1,140 

5,062,484,593 

14 

Fresno 

922,516 

114 

105,084,482 

7 

Houston 

4,177,646 

706 

2,948,039,040 

20 

Indianapolis 

1,607,486 

456 

733,470,541 

10 

Jersey  City 

608,975 

13,044 

7,943,237,618 

17 

Kansas  City,  MO-KS 

1,776,062 

329 

583,476,273 

13 

Las  Vegas,  NV-AZ 

1,563,282 

40 

62,076,079 

11 

Los  Angeles-Long  Beachd 

9,519,338 

2,344 

22,314,867,674 

40 

Louisville,  KY-IN 

1,025,598 

495 

507,651,616 

9 

Memphis,  TN-AR-MS 

1,135,614 

378 

428,953,952 

10 

Miami,  FL 

2,253,362 

1,158 

2,609,185,020 

19 

Milwaukee-Waukesha,  Wl 

1,500,741 

1,028 

1,542,728,464 

10 

Minneapolis-St.  Paul, 

2,968,806 

490 

1,453,687,745 

20 

MN-Wle 

New  Haven-Meriden,  CT 

542,149 

1,261 

683,670,545 

10 

New  Orleans 

1,337,726 

394 

526,405,217 

7 
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Table  4.1 — continued 


Urban  Areas 

Population3 

Population 
Density3  (per 
Square  Mile) 

Density- 

Weighted 

Population11 

FY2004  UASI 
Grant  Allo¬ 
cation'  ($ 
million) 

New  York,  NY 

9,314,235 

8,159 

75,991,762,554 

47 

Newark,  NJ 

2,032,989 

1,289 

2,619,713,383 

15 

Oakland,  CA 

2,392,557 

1,642 

3,927,449,645 

8 

Orange  County,  CAf 

2,846,289 

3,606 

10,262,626,470 

25 

Orlando 

1,644,561 

471 

774,794,778 

9 

Philadelphia,  PA-NJ 

5,100,931 

1,323 

6,749,136,215 

23 

Phoenix-Mesa 

3,251,876 

223 

725,649,640 

12 

Pittsburgh 

2,358,695 

510 

1,202,742,683 

12 

Portland-Vancouver, 

OR-WA 

1,918,009 

381 

731,703,925 

8 

Richmond-Petersburg 

996,512 

338 

337,254,906 

7 

Sacramento 

1,628,197 

399 

649,623,296 

8 

St.  Louis,  MO-IL 

2,603,607 

407 

1,060,496,877 

11 

San  Antonio 

1,592,383 

479 

762,291,362 

6 

San  Diego 

2,813,833 

670 

1,885,205,299 

10 

San  Francisco 

1,731,183 

1,705 

2,951,064,038 

26 

San  Jose 

1,682,585 

1,304 

2,193,476,169 

10 

Seattle-Bellevue-Everett 

2,414,616 

546 

1,318,032,823 

17 

Tampa-St.  Petersburg- 
Clearwater 

2,395,997 

938 

2,247,784,596 

9 

Washington, 

DC-MD-VA-WV 

4,923,153 

756 

3,723,526,125 

29 

NOTES:  (a)  Data  from  U.S.  Census  Bureau  (2000).  (b)  Calculated  as  population2/mile2. 
(c)  Data  from  DHS  (2004)  rounded  to  the  nearest  $  million,  (d)  Los  Angeles  and  Long 
Beach  received  separate  UASI  disbursements  though  are  considered  together  in  this 
analysis,  (e)  Minneapolis  and  St.  Paul  received  separate  UASI  disbursements  though 
are  considered  together  in  this  analysis,  (f)  Santa  Ana  and  Anaheim  received  separate 
UASI  disbursement  though  are  considered  together  in  this  analysis  as  Orange  County. 

However,  there  are  limitations  to  using  simple  indicators  of  ter¬ 
rorism  risk.  The  main  limitation  is  that  they  do  not  fully  reflect  the 
interactions  of  threat,  vulnerability,  or  consequences  discussed  in 
Chapter  Two.  As  a  result,  there  is  little  consensus  and  no  validated 
framework  for  deciding  how  to  use  several  simple  indicators  to  create 
a  single  risk  estimator.  For  example,  one  simple  indicator  of  risk  with 
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respect  to  biological  attacks  on  livestock  might  be  the  size  of  the  live¬ 
stock  population.  However,  there  is  no  theoretical  or  empirical  basis 
for  deciding  whether  counts  of  livestock  should  be  included  as  an 
agroterrorism  indicator  in  a  model  of  a  region’s  overall  risk,  and  if  so, 
in  what  proportion  to  other  indicators  like  population  or  energy  in¬ 
frastructure.  Furthermore,  including  an  indicator  like  livestock  could 
dramatically  alter  conclusions  about  the  distribution  of  risk  across 
regions.  In  short,  therefore,  stakeholders  with  concerns  about  differ¬ 
ent  types  of  terrorism  are  unlikely  to  agree  upon  any  model  of  risk 
that  relies  on  a  single  presumptive  risk  indicator. 


Event-Based  Models:  The  RMS  Terrorism  Risk  Model 

Event-based  models  are  built  upon  relatively  detailed  analysis  of  con¬ 
sequences  from  specific  attack  scenarios.  These  models  include  sensi¬ 
tivity  analysis  for  important  parameters  that  affect  consequences. 
They  may  include  components  to  model  multiple  types  of  events  and 
multiple  targets.  They  may  also  include  modules  that  translate  expert 
judgments  of  likelihood  or  consequences.  The  strength  of  event-based 
models  lies  in  the  greater  detail  they  enable  in  analysis.  The  weakness 
is  that  to  obtain  this  detail  analysts  must  estimate  many  uncertain  pa¬ 
rameters  that  define  the  attack  scenarios.  One  example  of  an  event- 
based  model  is  the  RMS  Terrorism  Risk  Model.  The  RMS  model  was 
developed  as  a  tool  for  the  insurance  and  reinsurance  industries  to 
assess  risks  of  macroterrorism. 

To  reflect  risk  as  a  function  of  threat,  vulnerability,  and  conse¬ 
quences,  the  RMS  model  calculates  the  expected  annual  consequences 
(human  and  economic)  from  diverse  terrorist  threats.  The  methodol¬ 
ogy  relies  on  models  of  specific  threat  scenarios,  calculations  of  eco¬ 
nomic  and  human  life  consequences  of  each  scenario,  and  assessments 
of  the  relative  probability  of  different  types  of  attacks  on  different  tar¬ 
gets.  The  RMS  model  calculates  the  threat  of  different  types  of  at¬ 
tacks  at  different  targets  using  expert  judgment  about  target  selection 
by  terrorists,  capabilities  for  different  attack  modes,  overall  likelihood 
of  attack,  and  propensity  to  stage  multiple  coordinated  attacks. 
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The  RMS  model  assesses  vulnerability  by  taking  into  account 
how  security  measures  may  lower  the  overall  level  of  threat  to  a  class 
of  targets  or  deflect  risk  from  one  target  to  another.  It  then  calculates 
the  consequences  of  terrorist  attacks  for  37  attack  modes  (see  Table 
4.2)  viewed  as  representative  of  the  types  of  events  that  terrorists  are 
capable  of  and  motivated  to  attempt.  Consequences  are  assessed  in 
terms  of  economic  losses,  injuries,  and  fatalities  using  geocoded  data¬ 
bases  of  population  density,  human  activity  patterns,  business  activi¬ 
ties,  and  values  of  buildings  and  their  contents. 

RMS  selected  the  attack  modes  to  be  sufficiently  distinct  and 
well  enough  defined  so  that  it  is  possible  to  specify  scenario  parame- 


Table  4.2 

Modes  of  Attack  Modeled  in  the  RMS  Terrorism  Risk  Model 


Attack  Mode 

Description  and  Alternatives 

Surface-to-air  missile 

Commercial  jet  airliner  shot  down 

Bomb 

600-lb;  one-ton;  two-ton;  five-ton;  or  10-ton 

Aircraft  impact 

Hijacked  commercial  jet  airliner  flown  into  a  target 

Conflagration 

9,000-gallon  gasoline  tanker  hijacked  and  set  on  fire 

Sabotage:  industrial,  explosion 

5-,  50-,  or  150-ton  trinitrotoluene  (TNT)  equivalent 

Sabotage:  industrial,  toxic 
release 

5%,  40%,  or  100%  of  Bhopal  accident 

Sabotage:  industrial,  explosion 
+  release 

5-ton  +  5%  Bhopal;  50-ton  +  40%  Bhopal;  or  150-ton 
+  100%  Bhopal 

Sabotage:  nuclear  plant, 
radiation  release 

0.5%;  5%;  or  20%  of  inventory 

Dirty  bomb,  cesium  137 

1,500  Curies  or  15,000  Curies 

Chemical,  sarin  gas 

Indoors:  10  kg;  Outdoors:  10  kg;  300  kg;  or  1,000  kg 

2%  anthrax  slurry  released 
outdoors 

1  kg,  10  kg,  or  75  kg  of  slurry 

Weaponized  anthrax  released 
indoors 

40  g  of  weaponized  anthrax 

Smallpox 

10,  100,  or  1,000  initially  infected 

Genetically  engineered  smallpox 

100  or  1,000  initially  infected 

Nuclear  bomb 

1  kiloton  or  5  kiloton 

SOURCE:  RMS,  Inc. 
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ters  so  that  the  losses  from  the  event  can  be  modeled.  Sensitivity 
analysis  is  used  to  estimate  the  expected  event  outcomes  given  a  range 
of  relevant  model  parameters  for  each  event  scenario.  For  example, 
models  using  plume  dispersion  estimates  take  account  of  variation  in 
wind  speed,  release  point  above  ground,  and  atmospheric  stability. 
However,  the  events  were  also  selected  to  represent  a  sufficiently  wide 
range  of  potential  attacks  to  cover  the  loss  potential  from  terrorism. 
While  some  may  view  this  list  to  be  comprehensive,  others  may  no¬ 
tice  that  attacks  involving  suicide  bombers,  liquefied  natural  gas 
(LNG)  tankers,  or  other  commonly  discussed  attack  modes  have  been 
omitted.  This  point  identifies  an  important  limitation  of  event-based 
models.  That  is,  results  are  dependent  on  a  large  number  of  under¬ 
lying  assumptions. 

The  RMS  Terrorism  Risk  Model  allows  analysts  to  test  the  ro¬ 
bustness  of  model  results  to  assumptions  through  parametric  analysis 
of  threat,  vulnerability,  and  consequences.  One  way  this  is  done  is 
through  use  of  elicited  threat  outlooks.  The  expert  elicitation  process 
used  by  RMS  has  produced  three  perspectives  on  terrorist  threat  for 
the  next  year:  a  standard,  enhanced,  and  reduced  threat  outlook.  All 
of  these  perspectives  incorporate  consideration  of  al  Qaeda  and  asso¬ 
ciated  groups,  other  foreign  threat  groups  including  Hizballah,  and 
domestic  terrorist  groups.1  Each  perspective  represents  an  aggregation 
of  different  beliefs  about  terrorist  motivations  and  capabilities. 

Table  4.3  provides  an  overview  of  the  assumptions  used  for  each 
of  the  included  threat  perspectives.  The  standard  outlook  assumes  a 
terrorist  threat  that  is  primarily  from  al  Qaeda,  though  potentially 
from  other  foreign  organizations,  with  al  Qaeda  having  a  low  likeli¬ 
hood  of  using  CBRN  weapons,  and  an  overall  low  likelihood  of  mul¬ 
tiple,  coordinated  attacks.  The  enhanced  outlook  reflects  a  greater 
likelihood  that  a  terrorist  attack  might  occur,  that  al  Qaeda  might  use 
CBRN  attacks,  and  that  attacks  would  involve  multiple,  coordinated 
events.  The  enhanced  outlook  also  has  a  higher  Threat  Reduction 


1  Domestic  terrorist  groups  considered  include  right-wing,  left-wing,  and  special  interest 
group  extremists. 
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Table  4.3 

How  Terrorism  Outlooks  in  the  RMS  Model  Reflect  Perspectives  on  Threat 


RMS  Risk 

Outlooks  2004 

Probability  of 
One  or  More 
Attacks  (Al 
Qaeda/Other 
Foreign) 

Probability  of 
Chemical, 
Biological, 
Radiological,  or 
Nuclear 
Terrorism  (Al 
Qaeda/Other 
Foreign) 

Multiplicity3  (Al 
Qaeda/Other 
Foreign) 

Threat  Reduction 
Factorb 

Standard 

33%/ 18% 

10%  /  0% 

Low  /  Very  Low 

0.5 

Enhanced  Threat 

40%/ 18% 

20%  /  0% 

Moderate  /  Very 
Low 

0.6 

Reduced  Threat 

33% /0% 

0%  /  0% 

Low  /  — 

0.5 

NOTES:  (a)  Adjusts  relative  probabilities  of  events  based  upon  the  probability  that 
coordinated  attacks  take  place,  (b)  Adjusts  the  relative  probabilities  given  that  security 
measures  make  subsequent  attacks  more  difficult  after  a  single  event  occurs  in  a  given 
year. 


Factor,  reflecting  that  with  a  heightened  awareness  of  terrorism  activi¬ 
ties,  security  measures  would  be  tighter  and  have  a  greater  effect  on 
reduction  of  secondary  attacks  after  a  single  attack  happens.  Finally, 
the  reduced  outlook  assumes  that  the  only  macroterrorist  threats  to 
the  United  States  are  those  posed  by  al  Qaeda. 

We  used  the  RMS  Terrorism  Risk  Model  to  calculate  expected 
annual  consequences  of  terrorist  attacks  (i.e.,  terrorism  risk).  Losses 
were  expressed  in  terms  of  numbers  of  fatalities,  numbers  of  injuries, 
and  total  property  damage  in  dollars  (buildings,  building  contents, 
and  business  interruption).  Impacts  were  then  aggregated  across  the 
urban  areas  that  received  funding  through  the  UASI  grant  program 
by  summing  the  expected  annual  consequences  for  each  of  the  attack¬ 
mode  target  pairs  modeled  for  an  urban  area. 

Calculating  an  Aggregated  Risk  Estimator 

By  considering  three  perspectives  on  threat  (the  RMS  standard,  en¬ 
hanced,  and  reduced  threat  outlooks),  the  RMS  results  provide  three 
perspectives  on  terrorism  risk  for  the  UASI-funded  urban  areas.  Re¬ 
sults  from  the  RMS  model  are  shown  in  Table  4.4.  Though  50  urban 


Table  4.4 

Estimates  of  Expected  Annual  Terrorism  Consequences  in  UASI  Urban  Areas 
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Orange  County,  CAC  3.7  1.0  6.7  2  0.01  6  41  0.5  97 

Orlando  0.6  0.4  0.9  0.1  0.03  0.3  3  1  7 

Philadelphia,  PA-NJ  21  8  28  9  5  13  190  92  273 


Combined  Property  Total 
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NOTE:  (a)  Los  Angeles  and  Long  Beach  received  separate  UASI  disbursements  though  are  considered  together  in  this  analysis, 
(b)  Minneapolis  and  St.  Paul  received  separate  UASI  disbursements  though  are  considered  together  in  this  analysis,  (c)  Santa  Ana  and 
Anaheim  received  separate  UASI  disbursements  though  are  considered  together  in  this  analysis  as  Orange  County. 
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areas  were  allocated  UASI  funding  in  FY2004,  several  of  these  were 
analyzed  as  larger  urban  areas  because  of  how  the  RMS  model  is  con¬ 
figured.  Specifically,  Los  Angeles  and  Long  Beach,  Santa  Ana  and 
Anaheim,  and  Minneapolis  and  St.  Paul  received  separate  allocations 
but  were  modeled  as  Los  Angeles-Long  Beach,  Orange  County,  and 
Minneapolis-St.  Paul,  respectively.  As  a  result,  the  analysis  covers  47 
urban  areas  instead  of  50.  Subsequent  discussions  only  use  risk  esti¬ 
mates  derived  from  expected  fatalities.  However,  the  data  in  this  table 
indicates  that  for  results  from  the  RMS  model,  expected  fatalities, 
injuries,  and  economic  losses  are  highly  correlated  with  each  other. 

For  the  purpose  of  estimating  each  region’s  overall  risk,  we  must 
select  a  single  risk  estimate  for  each  city.  Moreover,  we  would  like  the 
overall  risk  estimate  to  characterize  risk  well,  regardless  of  which  of 
the  three  threat  perspectives  proves  to  be  closest  to  true  risk.  As  dis¬ 
cussed  in  Chapter  Three,  doing  so  requires  combining  multiple  esti¬ 
mates  of  terrorism  risk.  We  calculated  an  aggregated  risk  estimator 
from  the  three  sets  of  expected  fatalities  generated  by  the  RMS 
model,  c--,  using  a  constrained  optimization.  The  optimization  iden¬ 
tified  a  set  of  nonnegative  city  shares  of  total  risk,  rp  that  sum  to  one, 
and  minimize  the  objective  function, 

£  'LIv(cv-r,f 

»'= 1  7=1 

where  A  is  an  indicator  variable  taking  the  value  1  if  ct)  >  r.  (i.e.,  risk  is 
underestimated)  or  else  the  value  0.  This  functional  form  of  objective 
function  used  in  this  analysis  incorporates  several  assumptions. 

First,  the  objective  function  above  minimizes  underestimation 
error  as  opposed  to  overall  error.  This  is  based  on  our  judgment  that 
is  better  to  minimize  potential  losses  from  terrorism  that  could  result 
from  underestimating  risk  than  it  is  to  ensure  that  each  city  has  an 
equal  chance  of  having  its  risk  over-  or  underestimated.  This  assump¬ 
tion  tends  to  favor  regions  that  have  a  distribution  of  consequences 


1  i  designates  the  urban  area  and  j  designates  the  threat  estimate  used. 
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with  a  tail  representing  very  large  consequences.  Thus,  risk  estimates 
for  dense,  urban  areas  will  be  larger  using  this  approach  than  an  ordi¬ 
nary  sum  of  squares  minimization.  Increasing  estimates  for  cities  like 
New  York  means  that  the  larger  cities  to  do  not  bear  a  disproportion¬ 
ate  share  of  potential  risk  underestimation.  Another  reasonable  ap¬ 
proach  would  be  to  minimize  overall  sum  of  squares  with  an  objective 
function  that  set  A  in  the  equation  above  to  a  constant  value  of  1. 
While  this  objective  function  is  better  understood  mathematically,  it 
is  not  more  desirable  normatively.  The  subsequent  analysis  only  pre¬ 
sents  results  using  the  objective  function  minimizing  underestimation 
error.  Analysis  using  an  overall  error  objective  function  leads  to  quali¬ 
tatively  similar,  though  not  identical  results. 

Second,  using  squared  error  assumes  that  larger  errors  are  much 
worse  than  smaller  errors.  If  underestimation  error  is  linked  to  pre¬ 
ventable  fatalities  or  other  damages,  this  assumption  reflects  realities 
about  risk  perceptions  of  catastrophic  events. 

Finally,  this  analysis  only  accounts  for  risk  as  measured  by  ex¬ 
pected  fatalities.  As  discussed  previously,  a  complete  treatment  of  ter¬ 
rorism  risk  estimation  must  use  methods  from  the  literatures  on 
multiobjective  decisionmaking  to  provide  transparency  into  value 
judgments  regarding  balancing  across  different  types  of  consequences. 


Comparison  of  Population,  Density-Weighted  Population, 
and  the  Aggregated  Estimate  of  Urban  Area  Risk  Shares 

Figure  4.1  compares  estimates  of  urban  area  shares  of  risk  derived 
from  our  aggregated  estimator,  and  two  commonly  used  indicators, 
population  and  density-weighted  population.  For  comparison,  the 
shares  of  DFiS  FY2004  UASI  allocations  are  also  included  in  this  fig¬ 
ure,  along  with  a  vertical  line  representing  equal  shares  across  all 
funded  urban  areas.  A  table  listing  the  data  in  Figure  4.1  can  be 
found  in  Appendix  A. 

Shares  of  total  population  across  the  UASI-funded  urban  areas 
are  presented  as  filled  circles.  The  size  of  city  shares  of  risk  using  this 


Urban  Areas 
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Figure  4.1 

City  Shares  of  Total  Risk  Estimated  Using  Four  Indicators  of  Risk,  Sorted  by 
Aggregated  Estimate,  with  a  Vertical  Line  Indicating  Equal  Risk  Across  Cities 


New  York  NY 
Chicago  IL 
Washington  DC-MD-VA-WV 
San  Francisco  CA 
Los  Angeles-Long  Beach  CA 
Boston  MA-NH 
Philadelphia  PA-NJ 
Houston  TX 
Newark  NJ 
Seattle-Bellevue-Everett  WA 
Orange  County  CA 
Jersey  City 
Detroit  Ml 
Dallas  TX 
Orlando  FL 
Atlanta  GA 
Oakland  CA 
San  Diego  CA 
Phoenix,  Mesa  AZ 
St.  Louis  MO-IL 
Cincinnati  OH-KY-IN 
Baltimore  MD 
Indianapolis  IN 
New  Haven-Meriden  CT 
Fresno  CA 
Columbus  OH 
New  Orleans  LA 
Sacramento  CA 
Buffalo-Niagara  Falls  NY 
Denver  CO 
Albany-Schenectady-Troy  NY 
Memphis  TN-AR-MS 
Cleveland-Lorain-Elyria  OH 
Las  Vegas  NV-AZ 
Richmond-Petersburg  VA 
San  Antonio  TX 
Baton  Rouge  LA 
Charlotte  -Gastonia-Rock  Hill  NC-SC 
Milwaukee-Waukesha  Wl 
Tampa-St.  Petersburg-Clearwater  FL 
Portland-Vancouver  OR-WA 
Louisville  KY-IN 
Miami  FL 
San  Jose  CA 
Kansas  City  MO-KS 
Minneapolis-St  Paul  MN-WI 
Pittsburgh  PA 
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measure  ranges  from  a  high  of  0.076  of  total  risk  (Los  Angeles-Long 
Beach,  CA)  to  a  low  of  0.004  (New  Haven-Meriden,  CT),  with  14 
metropolitan  areas  having  shares  greater  than  the  equal-share  line. 

Density-weighted  population  shares  (filled  squares  in  Figure  4.1) 
run  from  a  high  of  0.378  (New  York)  to  0.0003  (Las  Vegas),  thus 
resulting  in  a  much  larger  spread  of  estimated  shares  of  total  risk  than 
derived  by  the  population  estimator.  Moreover,  using  density- 
weighted  population,  just  eight  cities  are  found  to  have  more  than  the 
equal-share  allocation  of  terrorism  risk. 

The  aggregated  estimates  of  city  risk-shares  are  displayed  in  Fig¬ 
ure  4.1  as  open  diamonds.  Immediately  apparent  is  that  most  of  these 
estimates  of  city  risk  shares  are  several  orders  of  magnitude  lower  than 
the  population  or  density-weighted  population  estimates.  The  aggre¬ 
gated  estimates  range  from  7.87E-8  (Pittsburgh)  to  0.672  (New 
York),  with  just  six  cities  having  shares  greater  than  the  equal  share. 
Interestingly,  by  more  equally  distributing  underestimation  risk,  the 
aggregated  estimator  causes  32  of  the  UASI  urban  areas  to  be  counted 
as  having  virtually  no  share  of  total  terrorism  risk.  If,  for  instance,  the 
$675  million  FY04  UASI  funds  had  been  distributed  in  proportion 
to  the  aggregate  estimate  of  city  risk,  these  cities  would  have  received 
less  than  $70,000  in  total. 

Most  of  the  concentration  in  risk  found  with  the  aggregated  es¬ 
timator  is  due  to  similar  concentrations  in  the  RMS  estimates  of  ex¬ 
pected  annual  terrorism  consequences  for  each  metropolitan  area. 
Across  threat  outlooks,  New  York  has  an  average  of  58  percent  of  to¬ 
tal  expected  losses  in  the  RMS  model,  and  six  of  the  metropolitan 
areas  account  for  more  than  95  percent  of  total  losses.  Clearly,  the 
aggregated  estimator  is  sensitive  to  this  underlying  model  of  terrorism 
risk. 

Alternatively,  other  terrorism  risk  models  that  reflect  threat  and 
vulnerability  information  not  captured  in  the  RMS  model  would  be 
expected  to  yield  an  aggregated  estimator  that  suggests  less  concentra¬ 
tion  of  risk,  if  the  model  estimates  were  similarly  less  concentrated. 

Finally,  Figure  4.1  shows  how  FY2004  UASI  allocations  (open 
circles)  compare  with  risk  estimates.  Shares  of  UASI  funding  closely 
track  urban  areas’  shares  of  population.  On  average,  city  population 
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shares  differ  from  grant  allocation  shares  by  just  0.006,  with  the 
maximum  discrepancy  of  0.02  occurring  for  Los  Angeles.  If  one  be¬ 
lieves  the  underlying  assumptions  of  the  RMS  Terrorism  Risk  Model, 
then  the  distribution  of  resources  does  not  match  the  distribution  of 
terrorism  risk.  As  stated  previously,  this  might  be  acceptable  because 
of  issues  of  cost  effectiveness  of  available  risk  reduction  opportunities. 


CHAPTER  FIVE 

Evaluating  the  Performance  of  Different 
Estimates  of  Terrorism  Risk 


We  have  argued  that  uncertainties  in  the  distribution  of  terrorism  risk 
necessitate  risk  estimates  that  perform  well  across  a  range  of  assump¬ 
tions  about  terrorist  threat,  vulnerability,  and  consequences.  In  this 
chapter,  we  describe  a  model  for  evaluating  the  performance  of  alter¬ 
native  risk  estimates  across  a  range  of  plausible  terrorism  futures.  We 
use  this  model  to  compare  the  robustness  to  uncertainty  of  three  es¬ 
timates  introduced  earlier:  the  population,  density-weighted  popula¬ 
tion,  and  the  proposed  aggregated  estimate  as  indicators  of  the  share 
of  terrorism  risk  for  each  of  the  UASI  urban  areas. 


The  Performance  of  Terrorism  Risk  Estimates 

The  population,  density-weighted  population,  and  aggregated  esti¬ 
mates  each  offer  a  different  solution  to  the  problem  of  estimating  cit¬ 
ies’  unobservable  share  of  true  risk,  from  here  forward  referred  to  as  a 
city’s  risk  share.  Given  these  differences,  it  is  important  to  know 
which  indicator  offers  the  most  robust  depiction  of  true  risk.  As  we 
discussed  previously,  however,  we  are  less  concerned  with  the  prob¬ 
lem  of  indicators  overestimating  true  risk  than  we  are  with  underes¬ 
timation  of  true  risk;  therefore,  we  define  an  estimator’s  performance 
in  terms  of  how  well  it  minimizes  underestimation  of  each  urban 
area’s  terrorism  risk  share. 

Since  true  risk  shares  are  not  directly  observable,  we  conduct  a 
series  of  simulations  to  explore  the  performance  of  the  estimators 


37 


38  Estimating  Terrorism  Risk 


across  a  range  of  plausible  values  for  each  city’s  true  risk  share.  We 
begin  with  a  best-available  estimate  of  urban  areas’  true  risk  shares, 
and  then  systematically  allow  simulated  true  risk  to  vary  around  these 
best  estimates,  examining  how  each  of  the  risk-share  estimators  per¬ 
forms.  In  this  way,  we  examine  how  robust  the  different  estimators 
are  to  a  wide  range  of  plausible  futures  designed  to  represent  uncer¬ 
tainties  in  terrorist  motives,  targets,  methods,  and  capabilities  that  are 
fundamental  to  the  description  of  cities’  true  risk  shares. 

Two  sets  of  simulations  were  conducted.  In  the  first,  we  took 
RMS  estimates  of  expected  annual  consequences  for  each  city  as  the 
best  available  estimate  of  true  risk,  and  then  systematically  allow  for 
the  possibility  that  true  risk  may  differ  from  the  RMS  estimates  by  up 
to  two  orders  of  magnitude.  Thus,  for  instance,  if  the  RMS  estimate 
of  expected  annual  terrorism  fatalities  for  New  York  City  is  304,  we 
examine  the  performance  of  the  estimators  if  true  fatality  risk  for 
New  York  ranges  from  3.04  to  30,400.  Other  urban  areas’  true  risk 
shares  were  simultaneously,  and  independently,  allowed  to  fluctuate 
around  their  RMS  risk  estimates  by  up  to  two  orders  of  magnitude. 

Because  our  aggregated  estimator  is  derived  from  the  same  RMS 
estimates  of  city  risk,  this  first  simulation  may  lead  to  results  biased  in 
its  favor.  To  address  this  limitation,  we  conducted  a  second  simula¬ 
tion  in  which  each  estimator’s  performance  was  examined  after  as¬ 
suming  that  cities’  shares  of  true  risk  may  vary  by  up  to  two  orders  of 
magnitude  around  their  shares  of  total  density-weighted  population. 

Simulation  Methods 

The  simulations  are  designed  to  explore  a  range  of  possible  values  for 
true  city  risk  shares,  which  values  are  then  used  to  evaluate  the  per¬ 
formance  of  the  city  risk-share  estimators.  Each  simulated  true  risk 
trial  is  constructed  by  assuming  that  the  47  urban  areas,  i,  to  have  a 
set  of  true  risk  values,  Atj,  for  j  =  1,2,  ...  J  types  of  risk,  that  ran¬ 
domly  differ  from  some  best  estimates  of  that  city’s  risk,  Et~,  by  up  to 
a  factor  of  k\ 


a..  =  E.r 

V  V 
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where  u  is  drawn  from  the  uniform  random  distribution  U[-l,l]. 
Thus,  simulated  true  risk  may  range  from  Et-/  k  to  Etjk,  with  roughly 
half  of  all  simulations  falling  above  or  below  a  city’s  best  risk  esti¬ 
mate,  E. 

Simulated  true  absolute  risk  values  for  each  type  of  risk,  A--,  are 
converted  to  true  risk  shares,  R,  by  dividing  each  by  the  sum  of  all 
absolute  true  risk: 


R 


V 


In  the  first  series  of  simulations,  best  estimates  of  true  absolute 
risk  (E)  are  derived  from  the  RMS  estimates  of  expected  city  conse¬ 
quences  (injuries,  fatalities,  and  property  loss)  across  risk  outlooks 
(standard,  enhanced,  and  reduced  risk),  specifically: 

•  In  Model  la,  “All  Consequences,  All  Outlooks,”  three  types 
of  absolute  risk  are  averaged  to  construct  the  simulated  true 
risk  shares  (i.e.,  J  =  3).  These  consist  of  the  RMS  estimates  for 
expected  city  fatalities,  injuries,  and  property  loss  across  the 
three  risk  outlooks. 

•  In  Model  lb,  “All  Consequences,  Standard  Outlook,”  the 
J  =  3  types  of  absolute  risk  averaged  to  construct  simulated 
true  risk  shares  are  the  RMS  estimates  of  expected  city  fatali¬ 
ties,  injuries,  and  property  loss  for  the  standard  risk  outlook. 

•  In  Model  lc,  “All  Consequences,  Enhanced  Outlook,”  the 
J  =  3  types  of  absolute  risk  averaged  to  construct  simulated 
true  risk  shares  are  the  RMS  estimates  of  expected  city  fatali¬ 
ties,  injuries,  and  property  loss  for  the  enhanced  risk  outlook. 

•  In  Model  Id,  “All  Consequences,  Reduced  Outlook,”  the 
J  =  3  types  of  absolute  risk  averaged  to  construct  simulated 
true  risk  shares  are  the  RMS  estimates  of  expected  city  fatali¬ 
ties,  injuries,  and  property  loss  for  the  reduced  risk  outlook. 
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•  In  Model  le,  “Fatalities,  All  Outlooks,”  the  J  =  3  types  of  ab¬ 
solute  risk  averaged  to  construct  simulated  true  risk  shares  are 
the  RMS  estimates  of  expected  city  fatalities  under  the  three 
risk  outlooks. 

•  In  Model  If,  “Injuries,  All  Outlooks,”  the  J  =  3  types  of  ab¬ 
solute  risk  averaged  to  construct  simulated  true  risk  shares  are 
the  RMS  estimates  of  expected  city  injuries  under  the  three 
risk  outlooks. 

•  In  Model  lg,  “Property  Loss,  All  Outlooks,”  the  J  =  3  types  of 
absolute  risk  averaged  to  construct  simulated  true  risk  shares 
are  the  RMS  estimates  of  expected  city  fatalities  under  the 
three  risk  outlooks. 


In  Model  2,  density-weighted  population  is  used  as  the  absolute 
risk  estimate,  E,  to  construct  simulated  true  risk  shares. 

In  all  models,  5,000  trials  are  evaluated  for  each  of  seven  values 
of  k,  the  factor  by  which  true  risk  is  allowed  to  deviate  from  estimated 
absolute  risk  (£=1,  5,  10,  25,  50,  75,  and  100).  At  k=\,  simulated 
true  risk  shares  are  equal  to  the  best  estimates  of  true  risk  shares, 
whereas  at  £=100  simulated  true  risk  shares  deviate  from  the  esti¬ 
mated  true  risk  shares  by  up  to  a  factor  of  100. 

The  performance  of  each  of  the  risk-share  estimators,  m,  is 
evaluated  for  each  set  of  simulated  true  risk  shares  by  examining  the 
sum  of  squared  underestimation  error  across  cities: 

J  47 


7=1  i=l 

where  7  is  an  indicator  variable  taking  the  value  1  if  R~  >  Umi,  or  else 
the  value  0,  and  Umi  is  the  risk-share  estimate  for  city  i  made  by  risk- 
share  estimator  m.  Across  simulations,  we  also  examine  the  largest 
such  sum  of  squared  error  to  establish  the  worst  case  performance  of 
each  risk-share  estimator  at  each  value  of  k. 

In  addition  to  examining  the  performance  of  the  three  city  risk- 
share  estimators  for  each  of  the  two  surrogates  for  true  risk,  we  in- 
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elude  a  sixth  estimator  for  comparison  purposes.  This  random  risk- 
share  estimator  is  recalculated  in  each  simulation  trial,  and  estimates 
each  city’s  risk  share  as  a  random  value  drawn  from  a  uniform  distri¬ 
bution,  U[0,1],  divided  by  the  sum  of  the  uniform  values  drawn  for 
all  cities  in  the  current  trial. 

Simulation  Results 

Variability  in  city  shares  of  overall  risk.  The  simulations  are  de¬ 
signed  to  construct  a  range  of  plausible  values  for  each  of  the  UASI 
urban  areas’  shares  of  total  terrorism  risk,  so  that  the  robustness  to 
different  possible  futures  of  the  risk-share  estimators  can  be  com¬ 
pared.  To  illustrate  the  extent  of  variability  in  true  city  risk  shares 
produced  by  the  simulation,  we  examined  maximum,  minimum,  and 
quartile  risk  shares  for  each  city  across  the  5,000  simulation  trials  in 
Model  la  when  true  risk  was  allowed  to  vary  from  the  RMS  estimate 
of  risk  by  one  order  of  magnitude  ( k  =  10).  For  this  case,  when  true 
risk  differs  from  the  RMS  estimates  by  up  to  one  order  of  magnitude, 
share  of  total  true  risk  for  New  York  ranges  from  a  minimum  of 
0.028  to  a  maximum  of  0.961  of  all  UASI  urban  areas’  total  risk. 
Baton  Rouge,  in  contrast,  had  a  minimum  true  risk  share  of 
0.000002,  and  a  maximum  of  0.0027  of  all  cities’  risk.  Similar  vari¬ 
ability  in  simulated  true  risk-shares  was  observed  in  each  of  the  mod¬ 
els.  These  results  are  shown  in  Figure  5.1,  where  the  share  of  total 
true  risk  across  the  J  =  3  risk  types  for  city  i  is  calculated  as  follows: 

j  j 
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Figure  5.1 

Simulation  Results:  Distribution  of  City  Share  of  Total  True  Risk  When  True 
Risk  Deviates  from  RMS  Estimates  by  up  to  a  Factor  of  10,  Model  la 
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In  the  first  series  of  models,  simulated  true  risk  varies  around  the 
RMS  model  estimates  of  expected  fatalities,  injuries,  and  property 
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loss  across  terrorism  risk  outlooks.  As  such,  these  models  use  all  or  a 
superset  of  the  RMS  model  estimates  of  risk  as  the  basis  for  simulat¬ 
ing  true  risk  (three  types  of  consequences  in  each  of  three  terrorism 
risk  outlooks).  Since  the  aggregated  estimator  was  developed  to 
minimize  underestimation  error  using  the  RMS  model,  it  might  be 
expected  to  outperform  the  other  estimators.  Nevertheless,  we  include 
measurements  of  the  performance  of  the  aggregated  estimator  in  the 
first  series  of  models,  because  it  provides  information  on  how  well  an 
optimized  risk-share  estimator  could  perform,  which  aids  in  the  in¬ 
terpretation  of  the  performance  of  the  other  risk-share  estimators. 

Figure  5.2  presents  the  mean  underestimation  error  performance 
for  the  three  risk-share  estimators  and  the  random  estimator  when 
true  risk  is  assumed  to  vary  around  all  nine  RMS  estimates  of  city  ter¬ 
rorism  risk  (Model  la).  As  expected,  the  random  estimator  is  associ¬ 
ated  with  the  greatest  underestimation  error  and  the  aggregated  esti¬ 
mator  is  associated  with  the  lowest  underestimation  error. 

Figure  5.2 

Mean  Risk  Underestimation  as  True  Risk  Deviates  from  Estimates:  Model  la 
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Interestingly,  at  all  levels  of  k ,  the  density-weighted  population 
estimator  resulted  in  underestimation  error  closer  to  that  of  the  ag¬ 
gregated  estimator  than  to  that  of  the  random  or  population  estima¬ 
tors.  The  population  estimator,  in  turn,  had  underestimation  error 
that  was  consistently  closer  to  that  of  the  random  estimator  than  to 
that  of  either  the  density-weighted  population  or  robust  estimators. 

As  noted  earlier,  the  simulation  procedures  allowed  for  consider¬ 
able  variability  in  simulated  true  risk  estimates.  For  instance,  in  some 
trials  New  York  accounted  for  less  than  one  percent  of  total  risk, 
whereas  in  others  it  accounted  for  more  than  99  percent.  Given  this 
degree  of  uncertainty,  it  is  useful  to  examine  worst-case  scenarios  for 
the  various  city  risk-share  estimators. 

Figure  5.3  presents  the  maximum  underestimation  error  ob¬ 
served  across  5,000  simulations  for  each  of  the  risk-share  estimators  as 
the  variability  of  the  simulated  true  risk  increases.  In  this  worst-case 
analysis,  the  estimators  exhibit  the  same  rank  ordering  as  found  with 
mean  underestimation  error,  with  the  random  estimator  performing 
most  poorly  and  the  aggregated  estimator  performing  best.  However, 
the  maximum  underestimation  error  increases  only  modestly  for 
k  >  30. 

Figures  A.l  through  A. 3,  in  Appendix  A,  describe  estimator  per¬ 
formance  when  simulated  true  risk  is  based  on  just  the  RMS  stan¬ 
dard,  enhanced,  or  reduced  terrorism  threat  outlooks  (Models  lb,  lc, 
and  Id).  Figures  A.4  through  A.6,  in  Appendix  A,  describe  estimator 
performance  when  just  fatalities,  injuries,  or  property  loss  are  consid¬ 
ered  as  the  basis  for  simulating  different  true  risk  scenarios.  These 
models  test  the  performance  of  the  risk-share  estimators  across  alter¬ 
native  value  judgments  about  what  types  of  consequences  are  of 
greatest  concern.  Each  of  these  models  uses  just  a  subset  of  the  data 
used  to  calculate  the  robust  estimator,  and  again  it  outperforms  all 
other  city  risk-share  estimators.  Indeed,  the  relative  performance  of 
the  risk-share  estimators  in  each  of  these  models  is  similar  to  that 
found  in  Model  la. 
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Figure  5.3 

Maximum  Risk  Underestimation  as  True  Risk  Deviates  from  Estimates: 
Model  la 
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In  Model  2,  simulated  true  risk  is  based  on  density-weighted 
population,  rather  than  RMS  estimates  of  risk.  As  seen  in  Figure  5.4, 
this  change  has  the  effect  of  making  the  density-weighted  population 
estimator  the  optimal  choice,  at  least  when  true  risk  is  assumed  to 
differ  from  density-weighted  populations  by  no  more  than  a  factor  of 
five.  Interestingly,  however,  the  aggregated  estimator  exhibits  a  com¬ 
parable  mean  underestimation  error  to  the  density-weighted  popula¬ 
tion  estimator  for  higher  levels  of  k.  As  in  the  first  series  of  models, 
the  population  estimator  produces  underestimation  errors  closer  to 
the  random  estimator  than  to  either  the  density-weighted  population 
estimator  or  the  aggregated  estimator. 

Figure  5.5  presents  the  worst-case  performances  for  Model  2. 
Here  the  aggregated  estimator  clearly  exhibits  higher  underestimation 
error  than  the  density-weighted  population  estimator,  but  otherwise 
the  relative  performance  of  the  estimators  is  similar  to  what  has  been 
observed  in  all  earlier  models. 
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Figure  5.4 

Mean  Risk  Underestimation  as  True  Risk  Deviates  from  Estimates:  Model  2 
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Figure  5.5 

Maximum  Risk  Underestimation  as  True  Risk  Deviates  from  Estimates: 
Model  2 
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Discussion  of  simulation  results.  The  consistency  of  findings 
across  the  models  has  several  important  implications.  First,  the  aggre¬ 
gated  estimator  exhibits  as  low  or  lower  mean  underestimation  error 
than  all  other  estimators  considered  in  this  study.  As  discussed  previ¬ 
ously,  this  was  expected  in  the  first  series  of  models  in  which  the  same 
data  used  to  generate  the  robust  estimator  were  used  to  simulate  true 
city  risk.  Nevertheless,  given  that  we  cannot  directly  observe  true  risk, 
we  must  estimate  relative  risk  based  on  some  best- available  model  of 
city  risk.  Our  models  show  that  using  procedures  like  those  we  devel¬ 
oped  to  produce  the  aggregated  estimator,  a  one-dimensional  estima¬ 
tor  can  be  calculated  that  minimizes  mean  underestimation  error 
across  multidimensional  model-based  risk  information,  and  which 
therefore  exhibits  lower  mean  underestimation  error  given  uncertain¬ 
ties  than  alternative  estimates  of  city  risk  shares.  Similarly,  even  when 
worst-case  scenarios  are  sought,  defined  as  those  simulated  allocations 
of  true  city  risk  that  maximize  estimator  error,  the  aggregated  estima¬ 
tor  exhibits  lower  underestimation  error  than  all  other  estimators,  or, 
in  the  case  of  Model  2,  performed  nearly  as  well  as  the  best  estimator. 

Second,  the  density-weighted  population  estimator  results  in 
substantially  reduced  underestimation  error  in  comparison  to  the 
random  and  population  estimators.  Moreover,  in  the  first  series  of 
models,  it  performed  nearly  as  well  as  the  aggregated  estimator,  and 
in  the  second  model,  it  was  equivalent  to  or  better  than  the  aggre¬ 
gated  estimator.  The  good  performance  of  the  density-weighted 
population  estimator  makes  it  particularly  attractive  for  problems  of 
establishing  terrorism  risk  shares  when  sophisticated  models  of  risk 
are  unavailable  or  when  there  is  a  need  for  a  simple  and  transparent 
risk  model,  as  might  be  the  case  should  such  models  need  to  be  pub¬ 
licly  debated. 

Third,  across  all  the  trials,  allocating  resources  using  the  popula¬ 
tion-based  approach  fares  little  better  than  the  random  estimator  that 
treats  Baton  Rouge  as  having  a  share  of  risk  equivalent  to  New 
York — or  flipping  a  coin.  The  random  estimator  consistently  per¬ 
formed  worse  than  all  other  estimators,  but  perhaps  not  as  differently 
as  might  be  expected.  This  equal  distribution  of  risk  shares  led  to  per¬ 
formance  not  far  worse  than  the  performance  of  the  population-based 
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approach.  This  finding  raises  important  questions  about  the  appro¬ 
priateness  of  population  as  a  risk  indicator. 

Finally,  the  aggregated  risk  estimator’s  performance  on  different 
unidimensional  risk  perspectives  is  a  measure  of  its  robustness.  In  this 
study,  the  relative  utility  of  the  different  city  risk-share  estimators 
does  not  depend  strongly  on  whether  or  not  terrorists  are  likely  to 
employ  weapons  of  mass  destruction.  That  is,  when  we  base  our 
simulation  on  true  risk  estimates  that  assume  terrorists  will  not  use 
weapons  of  mass  destruction  (the  RMS  Reduced  terrorism  outlook) 
in  Model  Id,  we  find  essentially  the  same  pattern  of  estimator  per¬ 
formance  as  when  we  base  the  simulation  on  estimates  of  relative  city 
risk  that  assume  that  the  risk  of  a  WMD  attack  is  even  greater  than 
suggested  by  current  conventional  wisdom  (the  RMS  Enhanced  ter¬ 
rorism  outlook).  We  find  a  similar  invariance  in  estimator  perform¬ 
ance  across  consequence  types  (fatalities,  injuries,  and  property  loss). 
For  the  models  used  in  this  study,  the  aggregated  estimator  is  rela¬ 
tively  robust  to  fluctuations  in  the  likelihood  of  WMD  attacks  and 
assumptions  that  prioritize  lives  or  property.  In  the  discussion  that 
follows,  these  observations  are  dependent  upon  the  models  used  in 
this  study  and  may  differ  if  other  models  are  considered.  Fiowever, 
the  approach  used  to  assess  robustness  would  remain  valid. 

To  summarize,  the  aggregated  estimator  performed  well  relative 
to  other  estimators,  providing  a  proof  of  concept  that  a  single,  one¬ 
dimensional  estimate  of  city  risk  shares  can  offer  a  solution  to  the 
problem  of  determining  cities’  relative  risks  that  can  be  robust  to  un¬ 
certainties  about  the  likelihood  of  WMD  attacks,  uncertainties  about 
which  consequences  ought  to  be  prioritized,  and  uncertainties  about 
the  expected  magnitude  of  risks  each  city  might  face.  Furthermore, 
the  density-weighted  population  estimator  fared  better  than  the  other 
simple  estimators  and  even  outperformed  the  aggregated  estimator  in 
some  cases. 

Limitations  of  the  modeling  exercise.  The  findings  in  this  chap¬ 
ter  are  subject  to  a  number  of  model  limitations,  several  of  which  we 
now  consider. 

Clearly,  the  first  series  of  models  reported  here  depend  on  the 
assumption  that  the  RMS  estimates  offer  reasonably  good  approxima- 
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tions  of  the  distribution  of  true  terrorism  risk  across  the  47  urban  ar¬ 
eas.  Different  assumptions  about  the  distribution  of  true  risk  could 
lead  to  different  conclusions  about  the  performance  of  the  different 
estimators.  For  instance,  the  RMS  model  describes  terrorism  risk 
largely  as  a  threat  to  urban  areas.  Models  that  provide  more  detail 
into  threats  to  rural  areas,  such  as  agroterrorism,  would  likely  provide 
different  observations.  However,  they,  too,  could  easily  be  incorpo¬ 
rated  into  the  methodology  demonstrated  in  this  study.  By  rerunning 
the  model  assuming  that  true  risk  is  distributed  proportionally  to 
density- weighted  population  (Model  2),  we  were  able  to  offer  a  lim¬ 
ited  test  of  the  sensitivity  of  our  results  to  our  reliance  on  RMS  esti¬ 
mates.  This  tests  a  case  where  true  risk  is  more  accurately  reflected 
with  a  simple  model  rather  than  a  complex  model  like  RMS.  A  more 
complete  test  would  compare  estimator  performance  when  true  risk  is 
simulated  using  an  independent  event-based  model  of  city  risk. 

Our  approach  to  simulating  uncertainty  in  true  risk  shares  and 
estimator  performance  relies  on  assumptions  that  may  be  incorrect  or 
inadequate.  In  simulating  true  city  risk  shares,  for  instance,  we  alter 
each  city’s  RMS  risk  estimate  by  a  factor  ku,  where  u  is  a  random  vari¬ 
able  with  a  uniform  [-1,1]  distribution.  This  approach  ensured  that 
approximately  half  of  all  trials  would  have  simulated  true  risk  values 
lower  than  the  RMS  estimates,  and  half  would  have  larger  values.  But 
alternative  distributions  for  the  random  variable  and  alternative  ap¬ 
proaches  to  modeling  variation  in  true  risk  are  possible  and  might 
lead  to  different  conclusions. 

To  explore  the  sensitivity  of  our  results  to  assumptions  about  the 
distribution  of  u  we  reran  Models  la  through  lg  using  a  range  of  al¬ 
ternative  distributions  for  u,  including  a  U-shaped  symmetric  distri¬ 
bution  (in  which  u  was  drawn  from  a  Beta  distribution  with  al¬ 
pha^. 2,  beta=0.2  and  which  ranged  from  -1  to  1,  B[0.2,0.2;-l,  If), 
an  inverse  U-shaped  symmetric  distribution  (B[10, 10,-1, 1J),  and 
asymmetric  distributions  skewed  toward  -1  or  1,  {B[0.2,5,-l,l]  and 
B[5,0.2,-l,l]).  In  each  of  these  model  series,  the  relative  performance 
of  the  risk-share  estimators  for  both  mean  underestimation  error  and 
maximum  underestimation  error  was  identical  to  those  reported 
above  when  using  a  uniform  distribution  for  u. 
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Another  modeling  assumption  we  made  was  that  the  harms  as¬ 
sociated  with  underestimating  city  risk  shares  are  not  linearly  related 
to  the  magnitude  of  the  underestimation  but  rather  grow  exponen¬ 
tially  with  underestimation  errors.  As  such,  we  measured  estimator 
performance  by  summing  the  square  of  underestimation  error,  rather 
than,  for  instance,  the  sum  of  underestimation  errors  or  other  aggre¬ 
gations  of  error.  This  decision  treats  a  risk  underestimation  error  of 
0.2  for  some  city  as  substantially  worse  than  underestimation  by  0.1 
at  two  cities,  for  example.  Although  this  assumption  appears  reason¬ 
able,  the  true  relationship  between  underestimation  errors  and  harms 
may  be  different  and  not  well  described  by  squared  error.  A  more 
complete  test  would  compare  estimator  performance  when  error  is 
assumed  to  be  the  absolute  deviations  instead  of  squared  deviations. 


CHAPTER  SIX 

Conclusions  and  Recommendations 


To  improve  the  allocation  of  homeland  security  resources  and  thereby 
to  reduce  loss  of  life  and  property  to  terrorism  or  minimize  poor  in¬ 
vestments  in  homeland  security  measures  if  attacks  do  not  take  place, 
it  is  essential  to  have  good  estimates  of  the  terrorism  risk  to  which 
different  regions  or  groups  are  exposed.  This  objective  has  been  diffi¬ 
cult  to  achieve  for  many  reasons,  including  confusion  about  the  defi¬ 
nition  of  risk  and  the  absence  of  a  systematic  framework  for  selecting 
risk  indicators.  This  monograph  offers  a  definition  of  risk  and  dis¬ 
cusses  the  relationships  among  threats,  vulnerabilities,  consequences, 
and  risk.  In  addition,  it  suggests  a  method  for  constructing  a  single¬ 
dimensional  estimate  of  city  risk  shares,  designed  to  perform  well 
across  a  wide  range  of  threat  scenarios,  risk  types,  and  other  sources  of 
uncertainty.  Finally,  it  proposes  and  demonstrates  a  framework  for 
comparing  the  performance  of  alternative  risk  estimates  given  uncer¬ 
tainty  in  terrorists’  intentions  and  capabilities,  target  vulnerabilities, 
and  the  likely  consequences  of  successful  terrorist  attacks. 


Defining  Terrorism  Risk 

Estimating  or  measuring  terrorism  risk  is  incomplete  without  a 
framework  that  considers  threat,  vulnerability,  and  consequences.  To 
establish  a  specific  and  actionable  framework  for  analysis  we  develop 
definitions  for  threats  as  external,  dynamic  forces  acting  on  targets  or 
infrastructure  and  vulnerabilities  as  properties  of  the  targets  them- 
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selves.  Together  threat  and  vulnerability  define  the  probability  that 
specific  types  of  damage-causing  attacks  will  occur  at  specific  targets 
during  specified  periods.  The  methods  used  to  estimate  threat  are 
qualitatively  different  than  those  used  to  measure  vulnerability  and 
consequences.  The  former  relies  on  collection  and  interpretation  of 
intelligence.  The  latter  requires  scientific  and  engineering  expertise  of 
attack  modes  and  target  responses  to  attacks.  Finally,  threat,  vulner¬ 
ability,  and  consequences  are  interdependent  since  terrorism  and 
homeland  security  is  a  multisided  game,  and  terrorists  may  act  strate¬ 
gically  to  increase  their  effectiveness.  Thus,  interactions  between 
threat,  vulnerability,  and  consequences  have  important  consequences 
for  risk  management,  though  not  explicitly  discussed  in  these  models. 


Reflecting  Uncertainty  in  Terrorism  Risk  Assessment 

Effective  assessment  of  terrorism  risks  requires  measures  that  can  ac¬ 
commodate  the  uncertainty  inherent  in  the  problem.  We  have  pro¬ 
posed  a  method  for  estimating  terrorism  risk  that  hedges  against  un¬ 
certainties  in  threats,  vulnerabilities,  and  consequences.  Similar  to 
practices  used  in  economic  forecasting,  this  estimator  aggregates  in¬ 
formation  from  multiple  models  or  experts.  The  aggregated  estimator 
is  robust  in  that  it  reflects  a  range  of  assumptions  about  terrorist 
threat  and  perspectives  on  relative  importance  of  different  measures 
of  consequences.  Further,  we  generate  this  aggregated  estimator  using 
an  approach  that  minimizes  the  extent  to  which  risk  is  underesti¬ 
mated  across  urban  areas.  Since  aggregation  can  mask  important  dif¬ 
ferences  between  models  or  experts,  it  is  important  to  consider  how 
this  aggregated  estimate  differs  from  estimates  based  on  single  per¬ 
spectives. 


Simple  Versus  Event-Based  Risk  Estimates 

We  describe  and  compare  two  approaches  frequently  used  to  estimate 
risk:  simple  indicators  and  event-based  models.  Examples  of  simple 
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indicators  include  population  and  density-weighted  population.  The 
RMS  Terrorism  Risk  Model  is  presented  as  an  example  of  event- 
based  models.  We  argue  that  event-based  models  offer  a  framework 
for  calculating  risk  that  overcomes  some  of  the  arbitrariness  of  simple 
indicators  of  risk  that  rely  on  presumptive  correlate  relationships. 

Supporting  this  claim  requires  an  approach  to  evaluation  that 
accommodates  several  types  of  uncertainty.  In  Chapter  Five,  we  dem¬ 
onstrated  a  simulation  method  for  evaluating  the  relative  performance 
of  alternative  risk  estimates  across  plausible  terrorism  futures.  This 
method  provides  insights  into  both  distinctions  between  risk  estima¬ 
tors  and  the  variance  of  single  estimators  to  assumptions  about  threat, 
vulnerability,  and  consequences. 

This  analysis  showed  that  our  aggregated  estimator  exhibited  as 
low  or  lower  underestimation  error  than  risk  estimates  based  on 
population  and  density-weighted  population,  demonstrating  that  a 
single  estimate  of  city  risk  shares  can  offer  a  solution  to  the  problem 
of  determining  cities’  shares  of  total  risk  that  is  robust  to  a  wide  range 
of  plausible  terrorism  risk  futures. 

The  density-weighted  population  estimator  results  in  substan¬ 
tially  reduced  underestimation  error  in  comparison  to  the  random 
and  population  estimators.  In  many  cases,  density-weighted  popula¬ 
tion  performs  comparably  to  the  robust  estimator.  This  suggests  that 
for  some  purposes,  use  of  density-weighted  population  as  a  simple 
risk  indicator  might  be  preferred  when,  for  example,  they  are  in¬ 
forming  decisions  with  a  strategic  time  horizon  and  lead  time. 

Decisionmakers  may  wish  to  estimate  risk  to  inform  strategic  re¬ 
source  allocations,  operational  resource  allocations,  or  to  evaluate 
how  terrorism  risk  is  changing.  Strategic  allocations  differ  from  op¬ 
erational  allocations  in  terms  of  the  frequency  and  ease  with  which 
allocation  decisions  can  be  changed.  Homeland  security  resources 
might  be  allocated  strategically  when  they  are  expensive  or  infeasible 
to  move  or  change  once  committed.  For  example,  resources  directed 
toward  training  emergency  responders  are  constrained  by  the  time  it 
takes  to  complete  training  and  the  resistance  of  emergency  response 
personnel  to  frequent  relocation.  On  the  other  hand,  operational  al¬ 
locations  might  be  made  in  response  to  specific  intelligence  or  to  ad- 
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dress  short-term  vulnerabilities.  For  example,  the  Democratic  and 
Republican  National  Conventions  in  2004  created  new  vulnerabili¬ 
ties,  requiring  enhanced  security  resources  for  a  short  period. 

Density-weighted  population  is  more  appropriate  than  either 
population  or  random  estimators  for  informing  strategic  resource  al¬ 
locations.  The  underlying  data  are  easily  obtained  and  provide  credi¬ 
ble,  face-valid  indicators  of  risk.  Both  of  these  factors  increase  the 
utility  of  density-weighted  population  in  public  debates  about  re¬ 
source  allocations. 

Density-weighted  population,  however,  does  not  allow  deci¬ 
sionmakers  to  see  how  changes  in  threat  or  vulnerability  information 
affect  terrorism  risk.  For  example,  when  making  operational  resource 
allocations  or  evaluating  the  effectiveness  of  preparedness  programs, 
decisionmakers  need  to  understand  how  specific  countermeasures  re¬ 
duce  or  change  the  profile  of  terrorism  risk.  Similarly,  a  crude  indica¬ 
tor  like  density-weighted  population  would  offer  no  guidance  about 
how  city  risk  estimates  might  change  with,  for  instance,  new  intelli¬ 
gence  about  terrorist  targeting  or  capabilities  of  using  WMD  attacks. 
For  these  purposes,  more  detailed  event-based  models  of  terrorism 
risk  are  essential. 

In  this  study,  a  single  estimate  was  shown  to  be  robust  across 
uncertainties  about  the  likelihood  of  WMD  attacks,  uncertainties 
about  which  consequences  ought  to  be  prioritized  in  considerations 
of  city  risk,  and  uncertainties  about  the  expected  magnitude  of  risks 
each  city  might  face.  For  example,  an  important  observation  is  that 
the  risk  profile  of  the  47  cities  did  not  change  significantly  with  the 
variability  (or  absence)  of  threats  from  weapons  of  mass  destruction. 
While  the  primary  focus  of  this  study  was  not  to  estimate  precisely 
terrorism  risk  in  the  United  States,  this  observation  raises  questions 
about  the  distribution  over  many  areas.  In  particular,  one  question 
might  be  to  investigate  whether  risk  is  a  characteristic  a  region’s  infra¬ 
structure  or  population  that  is  relatively  stable  across  different  threats. 
If  so,  this  would  be  an  important  observation  when  it  comes  to  policy 
and  resource  decisions. 

Observations  about  the  variance  of  the  aggregated  risk  estimate 
are  dependent  upon  the  results  of  the  RMS  model  of  terrorism  risk 
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and  set  of  alternative  estimates  to  which  the  aggregated  estimate  was 
compared.  However,  the  proposed  methods  can  be  readily  imple¬ 
mented  with  new  data  sources  or  other  models.  Adding  new  informa¬ 
tion  to  that  provided  by  the  RMS  Terrorism  Risk  Model  would  pre¬ 
sumably  further  improve  the  robust  estimates. 


Recommendations 

The  framework  for  defining  terrorism  risk  and  the  analysis  presented 
in  this  monograph,  leads  us  to  five  recommendations  for  improving 
the  allocation  of  homeland  security  resources. 

1.  DHS  should  consistently  define  terrorism  risk  in  terms  of  ex¬ 
pected  annual  consequences.  Calculating  expected  annual  conse¬ 
quences  requires  accounting  for  threat,  vulnerability,  and  conse¬ 
quences.  Defining  terrorism  risk  in  these  terms  facilitates  the 
incorporation  of  risk  reduction  as  the  goal  of  homeland  security  pro¬ 
grams. 

2.  DHS  should  seek  robust  risk  estimators  that  account  for  un¬ 
certainty  about  terrorism  risk  and  variance  in  citizen  values.  Given 
the  tremendous  uncertainties  surrounding  terrorism  risk  assessment, 
it  is  prudent  to  plan  for  the  range  of  plausible  futures  that  may  play 
out.  Several  approaches  are  available  for  generating  estimates  of  city 
risk  shares  that  offer  robust  characterizations  of  risk  across  multiple 
uncertainties  and  perspectives  on  relative  values  of  different  conse¬ 
quences.  Our  approach  to  this  problem  ensures  that  underestimation 
error  is  minimized. 

3.  DHS  should  develop  event-based  models  of  terrorism  risk, 
like  that  used  by  RAND  and  RMS.  Measuring  and  tracking  levels  of 
terrorism  risk  is  an  important  component  of  homeland  security  pol¬ 
icy.  These  data  provide  insight  into  how  current  programs  are  re¬ 
ducing  risk  and  when  and  where  new  terrorist  threats  may  be  emerg¬ 
ing.  Only  event-based  models  of  terrorism  risk  provide  insight  into 
how  changes  in  assumptions  or  actual  levels  of  threat,  vulnerability, 
and  consequences  affect  risk  levels.  This  characteristic  is  important 
for  informing  operational  level  problems  such  as  deciding  which  secu- 
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rity  and  preparedness  programs  to  implement.  Furthermore,  event- 
based  models  overcome  a  principal  shortcoming  of  models  that  com¬ 
bine  diverse  risk  indicators:  They  provide  a  coherent,  defensible 
framework  for  selecting  and  combining  information  about  threats, 
vulnerabilities,  and  consequences. 

4.  Until  reliable  event-based  models  are  constructed,  density- 
weighted  population  should  be  preferred  over  population  as  a  sim¬ 
ple  risk  indicator.  Density-weighted  population  is  reasonably  corre¬ 
lated  with  the  distribution  of  terrorism  risk  across  the  United  States, 
as  estimated  by  event-based  models  like  the  RMS  Terrorism  Risk 
Model.  To  support  strategic  policy  decisions  when  the  effects  of  new 
countermeasures  or  recent  intelligence  are  not  in  question,  density- 
weighted  population  is  a  useful  indicator  in  lieu  of  event-based  mod¬ 
els.  In  contrast,  our  results  suggest  that  population  offers  a  remarka¬ 
bly  weak  indicator  of  risk,  not  much  superior  to  estimating  risk  shares 
at  random. 

5.  DHS  should  fund  research  to  bridge  the  gap  between  terror¬ 
ism  risk  assessment  and  resource  allocation  policies  that  are  cost  ef¬ 
fective.  Until  the  relationship  between  allocation  amounts  and  risk 
reduction  is  understood,  resource  allocation  decisions  will  not  be  op¬ 
timized  for  reducing  casualties  and  property  loss.  To  these  ends,  DFiS 
should  evaluate  the  performance  of  the  formula  used  to  assign  UASI 
grants  using  the  approach  presented  in  this  study.  Fiomeland  security 
efforts  will  be  greatly  improved  with  better  understanding  of  both  the 
resources  required  to  affect  a  range  of  countermeasures  and  the  risk 
reduction  achieved  by  affecting  those  countermeasures. 
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Supporting  Figures  and  Table 


Figure  A.1 

Mean  Risk  Underestimation  as  True  Risk  Deviates  from  Estimates:  Model  1b 


Factor  (k)  by  which  true  risk  may  differ  from  RMS  estimates  of  risk 
combining  the  RMS  Standard  threat  outlook  and  all  consequences 

RAND  MG388-A.  1 
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Figure  A. 2 

Mean  Risk  Underestimation  as  True  Risk  Deviates  from  Estimates:  Model  1c 


Factor  (k)  by  which  true  risk  may  differ  from  RMS  estimates  of  risk 
combining  the  RMS  Enhanced  threat  outlook  and  all  consequences 

RAND  MG388-A.2 


Figure  A. 3 

Mean  Risk  Underestimation  as  True  Risk  Deviates  from  Estimates:  Model  Id 


RAND  MG388-A.3 


Factor  (k)  by  which  true  risk  may  differ  from  RMS  estimates  of  risk 
combining  the  RMS  Reduced  threat  outlook  and  all  consequences 
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Figure  A. 4 

Mean  Risk  Underestimation  as  True  Risk  Deviates  from  Estimates:  Model  1e 


Factor  (k)  by  which  true  risk  may  differ  from  RMS  estimates  of  risk 
combining  all  threat  outlooks  and  fatalities  as  consequences 

RAND  MG388-A.4 


Figure  A. 5 

Mean  Risk  Underestimation  as  True  Risk  Deviates  from  Estimates:  Model  If 


Factor  (k)  by  which  true  risk  may  differ  from  RMS  estimates  of  risk 
combining  all  threat  outlooks  and  injuries  as  consequences 

RAND  MG388-A.S 
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Figure  A. 6 

Mean  Risk  Underestimation  as  True  Risk  Deviates  from  Estimates:  Model  1g 


Factor  (k)  by  which  true  risk  may  differ  from  RMS  estimates  of  risk 
combining  all  threat  outlooks  and  property  losses  as  consequences 

RAND  MG388-A.6 
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Table  A.1 

Shares  of  2003  DHS  UASI  and  City  Risk  Shares  Estimated  Using  Population, 
Density-Weighted  Population,  and  Aggregated  Estimator  Methods 


Risk-Share  Estimator 


Metro  Area  DHS  Allocation  Population  Dens.-Wt.  Pop.  Aggregated 


Albany-Schenectady-Troy,  0.0102  0.0071 

NY 


Atlanta,  GA 
Baltimore,  MD 
Baton  Rouge,  LA 
Boston,  MA-NH 
Buffalo-Niagara  Falls 

Charlotte-Gastonia-Rock 
Hill,  NC-SC 

Chicago,  IL 
Cincinnati,  OH-KY-IN 

Cleveland-Lorain-Elyria, 

OH 

Columbus,  OH 
Dallas,  TX 
Denver,  CO 
Detroit,  Ml 
Fresno,  CA 
Houston,  TX 
Indianapolis,  IN 
Jersey  City,  NJ 
Kansas  City,  MO-KS 
Las  Vegas,  NV-AZ 
Los  Angeles-Long  Beach 
Louisville,  KY-IN 
Memphis,  TN-AR-MS 
Miami,  FL 

Milwaukee-Waukesha, 

Wl 


0.0159 

0.0335 

0.0236 

0.0208 

0.0107 

0.0049 

0.0283 

0.0278 

0.0150 

0.0095 

0.0110 

0.0122 

0.0506 

0.0675 

0.0189 

0.0134 

0.0155 

0.0184 

0.0129 

0.0126 

0.0181 

0.0287 

0.0128 

0.0172 

0.0204 

0.0362 

0.0105 

0.0075 

0.0296 

0.0341 

0.0151 

0.0131 

0.0254 

0.0050 

0.0197 

0.0145 

0.0156 

0.0128 

0.0599 

0.0777 

0.0133 

0.0084 

0.0149 

0.0093 

0.0284 

0.0184 

0.0151 

0.0122 

0.0012 

1.08E-06 

0.0138 

6.55E-05 

0.0124 

1.69E-06 

0.0011 

6.15E-07 

0.0286 

2.22E-02 

0.0044 

1.15E-06 

0.0033 

5.71  E-07 

0.0673 

1.10E-01 

0.0040 

1.82E-06 

0.0093 

9.44E-07 

0.0038 

1.25E-06 

0.0010 

3.12E-04 

0.0059 

1.10E-06 

0.0252 

1.04E-03 

0.0005 

1.33E-06 

0.0147 

1.52E-02 

0.0037 

1.52E-06 

0.0396 

1.23E-03 

0.0029 

2.53E-07 

0.0003 

8.95E-07 

0.1111 

3.73E-02 

0.0025 

4.52E-07 

0.0021 

1.08E-06 

0.0130 

3.95E-07 

0.0077 

5.23E-07 
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Table  A.1 — continued 


Risk-Share  Estimator 

Metro  Area 

DHS  Allocation 

Population 

Dens.-Wt.  Pop. 

Aggregated 

Minneapolis-St.  Paul, 
MN-WI 

0.0298 

0.0242 

0.0072 

8.98E-08 

New  Haven-Meriden,  CT 

0.0143 

0.0044 

0.0034 

1.39E-06 

New  Orleans,  LA 

0.0106 

0.0109 

0.0026 

1.19E-06 

New  York,  NY 

0.0696 

0.0760 

0.3785 

6.72E-01 

Newark,  NJ 

0.0223 

0.0166 

0.0130 

6.36E-03 

Oakland,  CA 

0.0116 

0.0195 

0.0196 

7.79E-06 

Orange  County,  CA 

0.0376 

0.0232 

0.0511 

2.66E-03 

Orlando,  FL 

0.0130 

0.0134 

0.0039 

3.06E-04 

Philadelphia,  PA-NJ 

0.0342 

0.0416 

0.0336 

1.53E-02 

Phoenix-Mesa,  AZ 

0.0181 

0.0265 

0.0036 

2.27E-06 

Pittsburgh,  PA 

0.0178 

0.0192 

0.0060 

7.87E-08 

Portland-Vancouver,  OR- 
WA 

0.0121 

0.0156 

0.0036 

4.55E-07 

Richmond-Petersburg, 

VA 

0.0097 

0.0081 

0.0017 

8.61  E-07 

Sacramento,  CA 

0.0119 

0.0133 

0.0032 

1.18E-06 

St.  Louis,  MO-IL 

0.0160 

0.0212 

0.0053 

1.84E-06 

San  Antonio,  TX 

0.0094 

0.0130 

0.0038 

6.77E-07 

San  Diego,  CA 

0.0155 

0.0230 

0.0094 

2.52E-06 

San  Francisco,  CA 

0.0392 

0.0141 

0.0147 

4.78E-02 

San  Jose,  CA 

0.0148 

0.0137 

0.0109 

2.67E-07 

Seattle-Bellevue-Everett 

0.0245 

0.0197 

0.0066 

5.93E-03 

Tampa-St.  Petersburg- 
Clearwater,  FL 

0.0137 

0.0195 

0.0112 

4.62E-07 

Washington,  DC-MD-VA- 
WV 

0.0434 

0.0402 

0.0185 

6.23E-02 
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